[01:55:48] (03PS1) 10Gergő Tisza: Disable MediaViewer thumbnail URL guessing for private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479997 (https://phabricator.wikimedia.org/T212099) [03:32:07] PROBLEM - puppet last run on cp3049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:45:57] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 990.18 seconds [04:03:19] RECOVERY - puppet last run on cp3049 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:05:17] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 196.29 seconds [04:20:29] !log tstarling@deploy1001 Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 59s) [04:20:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:22:52] !log tstarling@deploy1001 Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 44s) [04:22:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:17:11] (03PS1) 10ArielGlenn: switch snapshot1005,6,7,9 to use php 7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480003 (https://phabricator.wikimedia.org/T211935) [06:17:39] (03PS1) 10Tulsi Bhagat: Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 [06:23:53] (03PS2) 10Tulsi Bhagat: Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) [06:24:25] (03PS1) 10Marostegui: db-eqiad.php: Depool db1123 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480005 (https://phabricator.wikimedia.org/T202167) [06:25:53] (03PS1) 10Marostegui: dbproxy1010: Depool labsdb1010 [puppet] - 10https://gerrit.wikimedia.org/r/480006 (https://phabricator.wikimedia.org/T86338) [06:26:15] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1123 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480005 (https://phabricator.wikimedia.org/T202167) (owner: 10Marostegui) [06:26:55] (03CR) 10Marostegui: [C: 03+2] dbproxy1010: Depool labsdb1010 [puppet] - 10https://gerrit.wikimedia.org/r/480006 (https://phabricator.wikimedia.org/T86338) (owner: 10Marostegui) [06:27:21] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1123 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480005 (https://phabricator.wikimedia.org/T202167) (owner: 10Marostegui) [06:28:00] !log Depool labsdb1010 - T86338 [06:28:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:28:03] T86338: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 [06:28:21] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1123 T86338 T202167 (duration: 00m 46s) [06:28:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:28:25] T202167: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 [06:28:58] (03CR) 10ArielGlenn: [C: 03+2] switch snapshot1005,6,7,9 to use php 7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480003 (https://phabricator.wikimedia.org/T211935) (owner: 10ArielGlenn) [06:29:16] (03PS2) 10ArielGlenn: switch snapshot1005,6,7,9 to use php 7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480003 (https://phabricator.wikimedia.org/T211935) [06:29:57] !log Enable replication consistency options on codfw masters - T211973 [06:30:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:30:00] T211973: Check GTID, consistency options and notifications across the fleet - https://phabricator.wikimedia.org/T211973 [06:33:12] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1123 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480005 (https://phabricator.wikimedia.org/T202167) (owner: 10Marostegui) [06:36:22] !log Enable GTID on db2034 (x1 master) [06:36:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:36:34] !log Enable GTID on db2034 (x1 master) - T211973 [06:36:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:36:37] T211973: Check GTID, consistency options and notifications across the fleet - https://phabricator.wikimedia.org/T211973 [06:48:14] !log Enable GTID on db2035 (s2 master) - T211973 [06:48:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:48:17] T211973: Check GTID, consistency options and notifications across the fleet - https://phabricator.wikimedia.org/T211973 [07:01:02] (03CR) 10Giuseppe Lavagetto: [C: 03+1] cache_text: Vary for PHP7 [puppet] - 10https://gerrit.wikimedia.org/r/478680 (https://phabricator.wikimedia.org/T206339) (owner: 10BBlack) [07:15:01] 10Operations, 10Release Pipeline: blubber template for nodejs should allow defining configuration files to copy to the container - https://phabricator.wikimedia.org/T211580 (10Joe) I think adding an ENV variable to service-runner is probably the way to go. [07:19:33] (03PS1) 10ArielGlenn: switch dump scripts on snapshots 1005,6,7,9 to use php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480007 (https://phabricator.wikimedia.org/T211935) [07:24:38] 10Operations, 10Beta-Cluster-Infrastructure, 10Wikidata, 10serviceops, and 4 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10Joe) [07:25:21] 10Operations, 10ops-eqiad, 10serviceops, 10HHVM: mw1272 crashed: Bad page map in process hhvm - https://phabricator.wikimedia.org/T211668 (10Joe) [07:26:21] 10Operations, 10Traffic, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Joe) [07:27:57] (03CR) 10ArielGlenn: [C: 03+2] switch dump scripts on snapshots 1005,6,7,9 to use php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480007 (https://phabricator.wikimedia.org/T211935) (owner: 10ArielGlenn) [07:28:53] <_joe_> apergos: \o/ [07:29:23] still one host left, I expect to do that in a week, it runs the misc scripts some of which still need testing [07:31:27] <_joe_> still, given how painful it was to try to use HHVM for dumps [07:31:36] <_joe_> I'm happy it's going flawless [07:32:12] I had to force some php extensino installs manually but that's really all [07:35:22] 10Operations, 10Performance-Team, 10TechCom, 10Core Platform Team (Session Management Service (CDP2)), and 4 others: Establish an SLA for session storage - https://phabricator.wikimedia.org/T211721 (10Joe) While I guess we should create a new ticket to talk about `MainObjectStash` is used and where to mig... [07:35:26] 10Operations, 10netops, 10User-Elukey: migrate netinsights from rhenium to sulfur - https://phabricator.wikimedia.org/T212011 (10faidon) a:05faidon→03MoritzMuehlenhoff Moritz was working on that. [07:36:08] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1123" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480008 [07:56:09] (03PS2) 10Muehlenhoff: Switch jamesur to volunteer NDA [puppet] - 10https://gerrit.wikimedia.org/r/479516 [07:58:04] !log Enable GTID on db2052 (s5 master) - T211973 [07:58:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:58:07] T211973: Check GTID, consistency options and notifications across the fleet - https://phabricator.wikimedia.org/T211973 [07:58:23] 10Operations, 10CirrusSearch, 10Discovery-Search, 10serviceops: Find an alternative to HHVM curl connection pooling for PHP 7 - https://phabricator.wikimedia.org/T210717 (10Joe) [07:59:06] (03CR) 10Muehlenhoff: [C: 03+2] Switch jamesur to volunteer NDA [puppet] - 10https://gerrit.wikimedia.org/r/479516 (owner: 10Muehlenhoff) [08:04:33] (03PS1) 10Elukey: profile::kafkatee::webrequest::ops: ensure archive directory [puppet] - 10https://gerrit.wikimedia.org/r/480012 (https://phabricator.wikimedia.org/T211883) [08:06:01] (03PS1) 10Muehlenhoff: Remove access fo nithum [puppet] - 10https://gerrit.wikimedia.org/r/480014 [08:06:26] 10Operations, 10serviceops: Add `supervised` option to redis configuration - https://phabricator.wikimedia.org/T212102 (10Joe) [08:12:40] (03CR) 10Elukey: [C: 03+2] profile::kafkatee::webrequest::ops: ensure archive directory [puppet] - 10https://gerrit.wikimedia.org/r/480012 (https://phabricator.wikimedia.org/T211883) (owner: 10Elukey) [08:13:29] (03PS2) 10Muehlenhoff: Remove access fo nithum [puppet] - 10https://gerrit.wikimedia.org/r/480014 [08:13:55] (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1123" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480008 (owner: 10Marostegui) [08:14:58] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1123" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480008 (owner: 10Marostegui) [08:15:13] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1123" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480008 (owner: 10Marostegui) [08:15:53] (03CR) 10Muehlenhoff: [C: 03+2] Remove access fo nithum [puppet] - 10https://gerrit.wikimedia.org/r/480014 (owner: 10Muehlenhoff) [08:15:57] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1123 T86338 T202167 (duration: 00m 45s) [08:16:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:16:02] T86338: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 [08:16:02] T202167: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 [08:16:15] !log Stop replication on labsdb1011 [08:16:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:18:21] (03PS1) 10Marostegui: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480024 (https://phabricator.wikimedia.org/T86338) [08:19:33] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480024 (https://phabricator.wikimedia.org/T86338) (owner: 10Marostegui) [08:19:54] (03PS2) 10Muehlenhoff: Remove Diamond from Swift proxies [puppet] - 10https://gerrit.wikimedia.org/r/479600 [08:20:44] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480024 (https://phabricator.wikimedia.org/T86338) (owner: 10Marostegui) [08:21:25] (03CR) 10Muehlenhoff: [C: 03+2] Remove Diamond from Swift proxies [puppet] - 10https://gerrit.wikimedia.org/r/479600 (owner: 10Muehlenhoff) [08:21:44] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1077 T86338 T202167 (duration: 00m 45s) [08:21:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:21:48] T86338: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 [08:21:49] T202167: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 [08:21:52] !log Deploy schema change on db1077 with replication (lag will be generated on labsdb:s3) T86338 T202167 [08:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:30:59] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Use standard version of plain-text GPL [cookbooks] - 10https://gerrit.wikimedia.org/r/460731 (owner: 10Legoktm) [08:31:43] (03CR) 10Urbanecm: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [08:32:30] (03CR) 10jerkins-bot: [V: 04-1] Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [08:33:47] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480024 (https://phabricator.wikimedia.org/T86338) (owner: 10Marostegui) [08:36:33] (03PS3) 10Mathew.onipe: wdqs: reduce hiera configs via profile defaults [puppet] - 10https://gerrit.wikimedia.org/r/479376 (https://phabricator.wikimedia.org/T210431) [08:39:16] (03CR) 10Giuseppe Lavagetto: [C: 04-2] "I think this is underambitious. We should create a comprehensive role that allows to configure a basic webserver (apache or nginx?);" [puppet] - 10https://gerrit.wikimedia.org/r/479575 (owner: 10Paladox) [08:41:21] (03PS1) 10Elukey: profile::kafkatee::webrequest::ops: add quickdatacopy [puppet] - 10https://gerrit.wikimedia.org/r/480025 (https://phabricator.wikimedia.org/T211883) [08:42:41] (03PS2) 10Elukey: profile::kafkatee::webrequest::ops: add quickdatacopy [puppet] - 10https://gerrit.wikimedia.org/r/480025 (https://phabricator.wikimedia.org/T211883) [08:43:52] (03CR) 10Elukey: [C: 03+2] profile::kafkatee::webrequest::ops: add quickdatacopy [puppet] - 10https://gerrit.wikimedia.org/r/480025 (https://phabricator.wikimedia.org/T211883) (owner: 10Elukey) [08:44:11] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480027 [08:45:15] (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480027 (owner: 10Marostegui) [08:45:54] (03CR) 10Giuseppe Lavagetto: [C: 03+2] php: Add support for php 7.3 [puppet] - 10https://gerrit.wikimedia.org/r/479144 (owner: 10Paladox) [08:46:06] (03PS6) 10Giuseppe Lavagetto: php: Add support for php 7.3 [puppet] - 10https://gerrit.wikimedia.org/r/479144 (owner: 10Paladox) [08:46:18] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480027 (owner: 10Marostegui) [08:47:13] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1077 T86338 T202167 (duration: 00m 45s) [08:47:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:47:23] T86338: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 [08:47:23] T202167: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 [08:47:55] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480027 (owner: 10Marostegui) [08:49:06] (03PS1) 10Elukey: Temporary assigning role::logging::webrequest::ops to oxygen [puppet] - 10https://gerrit.wikimedia.org/r/480028 (https://phabricator.wikimedia.org/T211883) [08:49:22] (03PS2) 10Elukey: Temporary assigning role::logging::webrequest::ops to oxygen [puppet] - 10https://gerrit.wikimedia.org/r/480028 (https://phabricator.wikimedia.org/T211883) [08:50:27] (03CR) 10Elukey: [C: 03+2] Temporary assigning role::logging::webrequest::ops to oxygen [puppet] - 10https://gerrit.wikimedia.org/r/480028 (https://phabricator.wikimedia.org/T211883) (owner: 10Elukey) [08:53:58] (03CR) 10Smalyshev: [C: 03+1] wdqs: reduce hiera configs via profile defaults [puppet] - 10https://gerrit.wikimedia.org/r/479376 (https://phabricator.wikimedia.org/T210431) (owner: 10Mathew.onipe) [09:01:51] !log stop kafkatee on oxygen and rsync /srv/log data to weblog1001 [09:01:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:02:09] (03PS1) 10Marostegui: Revert "dbproxy1010: Depool labsdb1010" [puppet] - 10https://gerrit.wikimedia.org/r/480030 [09:02:34] (03PS2) 10Marostegui: Revert "dbproxy1010: Depool labsdb1010" [puppet] - 10https://gerrit.wikimedia.org/r/480030 [09:03:43] (03CR) 10Marostegui: [C: 03+2] Revert "dbproxy1010: Depool labsdb1010" [puppet] - 10https://gerrit.wikimedia.org/r/480030 (owner: 10Marostegui) [09:04:24] !log Repool labsdb1010 - T86338 [09:04:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:04:27] T86338: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 [09:06:05] (03PS1) 10Marostegui: dbproxy1010: Depool labsdb1011 [puppet] - 10https://gerrit.wikimedia.org/r/480031 (https://phabricator.wikimedia.org/T86338) [09:06:25] (03PS1) 10Muehlenhoff: Remove Diamond from further roles [puppet] - 10https://gerrit.wikimedia.org/r/480032 (https://phabricator.wikimedia.org/T183454) [09:07:16] (03CR) 10Marostegui: [C: 03+2] dbproxy1010: Depool labsdb1011 [puppet] - 10https://gerrit.wikimedia.org/r/480031 (https://phabricator.wikimedia.org/T86338) (owner: 10Marostegui) [09:08:52] !log Depool labsdb1011 - T86338 [09:08:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:09:00] (03PS1) 10Elukey: profile::kafkatee::webrequest::ops: widen allowed data via rsync [puppet] - 10https://gerrit.wikimedia.org/r/480033 (https://phabricator.wikimedia.org/T211883) [09:14:18] (03CR) 10Mathew.onipe: "PCC proclaims NOOP: https://puppet-compiler.wmflabs.org/compiler1002/13974/" [puppet] - 10https://gerrit.wikimedia.org/r/479376 (https://phabricator.wikimedia.org/T210431) (owner: 10Mathew.onipe) [09:18:19] 10Operations, 10CirrusSearch, 10Discovery-Search, 10serviceops: Find an alternative to HHVM curl connection pooling for PHP 7 - https://phabricator.wikimedia.org/T210717 (10Joe) I see another problem here: say we do what makes sense and make MediaWiki connect to -say - `localhost:10001` for the search clu... [09:20:39] (03CR) 10DCausse: [C: 03+1] "noop https://puppet-compiler.wmflabs.org/compiler1002/13975/" [puppet] - 10https://gerrit.wikimedia.org/r/479376 (https://phabricator.wikimedia.org/T210431) (owner: 10Mathew.onipe) [09:32:00] (03CR) 10Cparle: [C: 03+1] "Looks good to me. I don't have +2 on this repo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479997 (https://phabricator.wikimedia.org/T212099) (owner: 10Gergő Tisza) [09:36:25] (03CR) 10DCausse: "I agree with Filippo, working at the java level seems to be a lot easier." [puppet] - 10https://gerrit.wikimedia.org/r/479395 (https://phabricator.wikimedia.org/T208215) (owner: 10Mathew.onipe) [09:41:22] (03CR) 10Filippo Giunchedi: [C: 03+1] "LGTM, cc Antoine" [puppet] - 10https://gerrit.wikimedia.org/r/479845 (https://phabricator.wikimedia.org/T210486) (owner: 10Cwhite) [09:42:21] (03CR) 10Filippo Giunchedi: "See inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/479843 (https://phabricator.wikimedia.org/T210486) (owner: 10Cwhite) [09:42:47] (03CR) 10Filippo Giunchedi: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/479772 (https://phabricator.wikimedia.org/T210486) (owner: 10Cwhite) [09:43:04] (03PS1) 10Marostegui: Revert "dbproxy1010: Depool labsdb1011" [puppet] - 10https://gerrit.wikimedia.org/r/480035 [09:44:05] (03CR) 10Filippo Giunchedi: [C: 04-1] "global search/replace gone awry" [puppet] - 10https://gerrit.wikimedia.org/r/479139 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [09:44:10] !log Enable GTID on s4 codfw master - T211973 [09:44:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:44:15] T211973: Check GTID, consistency options and notifications across the fleet - https://phabricator.wikimedia.org/T211973 [09:49:45] (03CR) 10Filippo Giunchedi: [C: 04-1] profile: enable statsd_exporter and add matching rules to logstash::collector (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/479353 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [09:50:47] (03CR) 10Filippo Giunchedi: [C: 03+1] profile: enable statsd_exporter and add matching rules to ores::worker [puppet] - 10https://gerrit.wikimedia.org/r/479563 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [09:51:32] (03PS8) 10Filippo Giunchedi: logstash: add new logging kafka consumer [puppet] - 10https://gerrit.wikimedia.org/r/476472 (https://phabricator.wikimedia.org/T205851) [09:55:08] 10Operations, 10Operations-Software-Development: Decommission script race condition - https://phabricator.wikimedia.org/T206448 (10MoritzMuehlenhoff) [09:55:52] !log remove debmonitor entries for restbase2001-restbase2006 [09:55:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:57:13] (03CR) 10Filippo Giunchedi: [C: 03+2] logstash: add new logging kafka consumer [puppet] - 10https://gerrit.wikimedia.org/r/476472 (https://phabricator.wikimedia.org/T205851) (owner: 10Filippo Giunchedi) [09:59:32] 10Operations, 10Phabricator, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10ema) 05Open→03Resolved a:03ema Yes @mmodell, now I can. Thank you! [10:02:38] (03PS2) 10Elukey: profile::kafkatee::webrequest::ops: widen allowed data via rsync [puppet] - 10https://gerrit.wikimedia.org/r/480033 (https://phabricator.wikimedia.org/T211883) [10:03:58] (03CR) 10Elukey: [C: 03+2] profile::kafkatee::webrequest::ops: widen allowed data via rsync [puppet] - 10https://gerrit.wikimedia.org/r/480033 (https://phabricator.wikimedia.org/T211883) (owner: 10Elukey) [10:07:57] (03CR) 10Vgutierrez: [C: 03+1] Extend testConfigServerRemoval test case. [debs/pybal] - 10https://gerrit.wikimedia.org/r/447770 (owner: 10Mark Bergsma) [10:14:35] (03PS2) 10Marostegui: Revert "dbproxy1010: Depool labsdb1011" [puppet] - 10https://gerrit.wikimedia.org/r/480035 [10:17:32] (03CR) 10Filippo Giunchedi: [C: 03+1] role::maps::{master,slave}: add tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/479669 (https://phabricator.wikimedia.org/T211970) (owner: 10Ema) [10:18:03] (03CR) 10Marostegui: [C: 03+2] Revert "dbproxy1010: Depool labsdb1011" [puppet] - 10https://gerrit.wikimedia.org/r/480035 (owner: 10Marostegui) [10:18:52] !log Repool labsdb1011 - T86338 [10:18:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:18:55] T86338: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 [10:19:16] (03PS2) 10Michael Große: Perform even more PHP constraint checks before falling back [mediawiki-config] - 10https://gerrit.wikimedia.org/r/478630 (https://phabricator.wikimedia.org/T209504) [10:19:42] (03PS8) 10Ema: role::maps::{master,slave}: add tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/479669 (https://phabricator.wikimedia.org/T211970) [10:21:01] (03CR) 10Ema: [C: 03+2] role::maps::{master,slave}: add tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/479669 (https://phabricator.wikimedia.org/T211970) (owner: 10Ema) [10:26:25] 10Operations, 10Citoid, 10Services (watching): Support meta tag refresh redirects in citoid to support elsevier's linking hub - https://phabricator.wikimedia.org/T204032 (10Mvolz) p:05High→03Normal [10:28:06] !log installing openssl security updates on stretch [10:28:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:29:12] (03CR) 10DCausse: "I may lack context here but I have no objections accessing relforge for testing/developping purposes. If the intention is to have a stable" [puppet] - 10https://gerrit.wikimedia.org/r/479567 (https://phabricator.wikimedia.org/T78705) (owner: 10Dduvall) [10:30:04] jan_drewniak: Your horoscope predicts another unfortunate Wikimedia Portals Update deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1030). [10:30:20] just wondering since some local people are wondering: will WMF enable Flow aka StructuredDiscussion if locals want it? [10:40:21] (03PS1) 10Faidon Liambotis: apt: add component/pyall to stretch-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/480041 [10:41:30] (03CR) 10Volans: [C: 03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/480041 (owner: 10Faidon Liambotis) [10:42:31] (03PS2) 10Faidon Liambotis: apt: add component/pyall to stretch-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/480041 [10:42:55] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/480041 (owner: 10Faidon Liambotis) [10:43:03] (03CR) 10Faidon Liambotis: [C: 03+2] apt: add component/pyall to stretch-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/480041 (owner: 10Faidon Liambotis) [10:45:04] 10Operations, 10Phabricator, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10Peachey88) a:05ema→03mmodell [10:49:10] moritzm, volans, elukey fyi: [10:49:14] *** 3.4.9-1+deb9u1 500 [10:49:14] 500 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/pyall amd64 Packages [10:49:18] *** 3.6.7-1~bpo9+1 500 [10:49:18] 500 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/pyall amd64 Packages [10:49:21] *** 3.7.1-1~bpo9+1 500 [10:49:23] 500 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/pyall amd64 Packages [10:49:39] yay! (I didn't see the ! log for them though :-P ) [10:50:01] do we !log reprepro include nowadays? [10:50:03] we didn't use to [10:50:22] (and that's why we have the hook that emails us about that) [10:50:29] https://wikitech.wikimedia.org/wiki/Reprepro#HOWTO [10:50:59] I've always done it and saw others do too, but not sure if it's required honestly [10:52:27] !log reprepro include python 3.4, 3.6, 3.7 to component/pyall (use with care) [10:52:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:55:48] volans: :) [10:56:00] :D [10:56:29] so, one can put "deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/pyall" on any stretch system [10:56:45] thanks a lot paravoid! [10:56:46] and then "apt install python3.{4,5,6,7}-venv" [10:57:56] and then "mkdir venv; for v in `seq 4 7`; do python3.$v -m venv/py3$v; done" [10:58:09] (or just "tox"0 [10:58:11] ) [10:58:51] paravoid: want to do the honor updating T191764 ? [10:58:51] T191764: CI: run tests with multiple Python3 versions - https://phabricator.wikimedia.org/T191764 [10:59:25] not sure if it helps, our CI is stuck in jessie land [10:59:42] it's another push towards stretch ;) [10:59:50] this is really great, we'll be able to use superset without backporting patches and/or creating frankestein repos [11:00:26] elukey: are you using any system (python) libraries? [11:00:47] 80.2% of the fleet is on Stretch, so CI on jessie is a problem IMHO [11:01:41] yes it is [11:02:03] I'm afraid hashar will just say "switch to docker" or something [11:03:08] (03PS2) 10Arturo Borrero Gonzalez: toolforge: declare sonofgridengime::submit_host from a central point [puppet] - 10https://gerrit.wikimedia.org/r/479736 [11:03:45] volans: task updated [11:04:46] thanks a lot! [11:08:28] (03PS1) 10Fsero: Modified control file and changelog [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480049 [11:10:35] (03PS2) 10Fsero: Modified control file and changelog [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480049 [11:10:38] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "LGTM, fix these two nitpicks though." (032 comments) [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480049 (owner: 10Fsero) [11:11:23] (03PS3) 10Fsero: Modified control file and changelog [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480049 [11:11:59] (03CR) 10Fsero: [V: 03+2 C: 03+2] "fixed nitpicks" [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480049 (owner: 10Fsero) [11:19:10] 10Operations, 10MediaWiki-Logging, 10Wikimedia-Logstash, 10Patch-For-Review: Move mediawiki to new logging infrastructure - https://phabricator.wikimedia.org/T211124 (10fgiunchedi) Mediawiki in beta is now logging on `localhost` using ceeformatter, mediawiki-related dashboards on https://logstash-beta.wmfl... [11:23:20] (03PS12) 10ArielGlenn: convert dump scripts to python3 [dumps] - 10https://gerrit.wikimedia.org/r/478702 (https://phabricator.wikimedia.org/T210989) [11:26:47] 10Operations, 10Release Pipeline: blubber template for nodejs should allow defining configuration files to copy to the container - https://phabricator.wikimedia.org/T211580 (10mobrovac) [11:27:00] 10Operations, 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10Ladsgroup) >>! In T210721#4784825, @WMDE-leszek wrote: > I take it on me. I've briefly talked about this t... [11:27:26] (03PS1) 10Filippo Giunchedi: Use logging pipeline for group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480052 (https://phabricator.wikimedia.org/T211124) [11:34:33] (03PS1) 10Reedy: Re-instate "Turn wikimedia.org docroot into symlink to standard-docroot" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480055 [11:34:37] (03PS2) 10Reedy: Re-instate "Turn wikimedia.org docroot into symlink to standard-docroot" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480055 [11:38:03] ores revision to rollback 1a3de73719b61c1be5f40e9f1a4cd64c0ce9dbbd [11:38:12] I'm going to do a quick deploy for ores [11:38:59] (03PS1) 10Elukey: role::statistics::gpu: include standard [puppet] - 10https://gerrit.wikimedia.org/r/480056 [11:39:16] (03PS1) 10Filippo Giunchedi: role: add rsyslog::udp_localhost_compat to mediawiki hosts [puppet] - 10https://gerrit.wikimedia.org/r/480057 (https://phabricator.wikimedia.org/T211124) [11:40:30] (03CR) 10Elukey: [C: 03+2] role::statistics::gpu: include standard [puppet] - 10https://gerrit.wikimedia.org/r/480056 (owner: 10Elukey) [11:42:04] (03CR) 10Zfilipin: [C: 04-1] "A jenkins job failed:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [11:48:20] !log ladsgroup@deploy1001 Started deploy [ores/deploy@18d3657]: T206333 T211267 [11:48:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:48:24] T211267: ORES precaching seem to not understand its own config - https://phabricator.wikimedia.org/T211267 [11:48:24] T206333: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 [11:49:45] 10Operations, 10Wikimedia-Logstash, 10User-fgiunchedi, 10User-herron: rack/setup/install centrallog1001.eqiad.wmnet - https://phabricator.wikimedia.org/T200706 (10fgiunchedi) a:05fgiunchedi→03None [11:49:52] Let's get the party started :D [11:51:06] (03PS1) 10Ema: maps: add kartotherian-ssl LVS service [puppet] - 10https://gerrit.wikimedia.org/r/480060 (https://phabricator.wikimedia.org/T211970) [11:55:15] jouncebot, next [11:55:15] In 0 hour(s) and 4 minute(s): European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1200) [11:59:26] (03PS1) 10Fsero: fix some bugs over changelog [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480062 [12:00:04] (03CR) 10Fsero: [V: 03+2 C: 03+2] "i'm automerging this since its a simple typo" [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/480062 (owner: 10Fsero) [12:00:04] addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: (Dis)respected human, time to deploy European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1200). Please do the needful. [12:00:04] Urbanecm and Tulsi: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [12:00:11] I can SWAT today [12:00:21] Urbanecm, Tulsi: around for swat? [12:00:24] * Urbanecm waves to zeljkof [12:01:11] Urbanecm: do your commits need testing? [12:01:42] first one doesn't, the second one does [12:01:47] Urbanecm: ok [12:02:34] !log ladsgroup@deploy1001 Finished deploy [ores/deploy@18d3657]: T206333 T211267 (duration: 14m 14s) [12:02:36] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "Seems sane." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/480060 (https://phabricator.wikimedia.org/T211970) (owner: 10Ema) [12:02:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:02:38] T211267: ORES precaching seem to not understand its own config - https://phabricator.wikimedia.org/T211267 [12:02:39] T206333: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 [12:03:42] zeljkof: hey, once you're done, please let me know, I have some stuff to deploy :D [12:03:45] PROBLEM - HHVM rendering on mwdebug1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:52] Amir1: sure [12:05:57] RECOVERY - HHVM rendering on mwdebug1002 is OK: HTTP OK: HTTP/1.1 200 OK - 78767 bytes in 0.247 second response time [12:06:10] (03CR) 10Zfilipin: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479379 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:06:43] (03PS3) 10Urbanecm: Update be.wikisource logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479713 (https://phabricator.wikimedia.org/T211795) (owner: 10Tulsi Bhagat) [12:06:45] (03PS3) 10Urbanecm: Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [12:07:17] (03Merged) 10jenkins-bot: Upload custom minerva logo for cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479379 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:07:33] zeljkof, ^^ I rebased Tulsi's patch to fix the jenkins error (hopefully) [12:07:48] (03CR) 10Alexandros Kosiaris: [C: 03+2] ci: Remove minikube from pipeline instances [puppet] - 10https://gerrit.wikimedia.org/r/479746 (owner: 10Thcipriani) [12:07:55] (03PS2) 10Alexandros Kosiaris: ci: Remove minikube from pipeline instances [puppet] - 10https://gerrit.wikimedia.org/r/479746 (owner: 10Thcipriani) [12:08:00] (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] ci: Remove minikube from pipeline instances [puppet] - 10https://gerrit.wikimedia.org/r/479746 (owner: 10Thcipriani) [12:08:29] Urbanecm: looks like they are not around anyway :/ [12:08:41] In that case, I can take care about their patches [12:08:54] thanks [12:08:55] !log zfilipin@deploy1001 Synchronized static/images/mobile/copyright/: SWAT: [[gerrit:479379|Upload custom minerva logo for cswiki (T210979)]] (duration: 00m 44s) [12:08:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:08:58] T210979: Deploy localized cswiki wordmark on mobile Wikipedia - https://phabricator.wikimedia.org/T210979 [12:08:58] yw zeljkof [12:09:12] Urbanecm: 479379 deployed, please test [12:09:15] thx [12:09:28] (03PS4) 10Zfilipin: Use new minerva logos for cswiki in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479963 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:09:38] zeljkof, lgtm [12:09:57] (03CR) 10Zfilipin: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479963 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:11:21] (03Merged) 10jenkins-bot: Use new minerva logos for cswiki in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479963 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:13:30] (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] "Thanks! That's the only use of that package, I 'll remove it from our repos" [puppet] - 10https://gerrit.wikimedia.org/r/479746 (owner: 10Thcipriani) [12:13:56] Urbanecm: 479963 at mwdebug [12:13:59] ack [12:14:41] zeljkof, lgtm, please deploy [12:14:47] ok [12:15:08] (03CR) 10jenkins-bot: Upload custom minerva logo for cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479379 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:15:10] (03CR) 10jenkins-bot: Use new minerva logos for cswiki in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479963 (https://phabricator.wikimedia.org/T210979) (owner: 10Urbanecm) [12:15:38] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:479963|Use new minerva logos for cswiki in IS.php (T210979)]] (duration: 00m 45s) [12:15:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:15:42] T210979: Deploy localized cswiki wordmark on mobile Wikipedia - https://phabricator.wikimedia.org/T210979 [12:16:06] Urbanecm: 479963 deployed, please test [12:16:21] zeljkof, lgtm, thanks [12:16:25] (03PS2) 10Ema: maps: add kartotherian-ssl LVS service [puppet] - 10https://gerrit.wikimedia.org/r/480060 (https://phabricator.wikimedia.org/T211970) [12:17:02] (03CR) 10Zfilipin: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [12:17:25] (03CR) 10Ema: [C: 03+2] maps: add kartotherian-ssl LVS service [puppet] - 10https://gerrit.wikimedia.org/r/480060 (https://phabricator.wikimedia.org/T211970) (owner: 10Ema) [12:17:56] (03CR) 10Zfilipin: Enable 'extendedmover' user group at ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [12:18:03] (03PS6) 10Zfilipin: Enable 'extendedmover' user group at ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [12:18:11] (03CR) 10Zfilipin: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [12:19:17] (03Merged) 10jenkins-bot: Enable 'extendedmover' user group at ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [12:20:09] Urbanecm: 479723 at mwdebug [12:20:15] ack [12:20:25] (03PS1) 10Volans: icinga: add execution of Icinga commands [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) [12:20:27] (03PS1) 10Volans: remote: add more functionalities [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) [12:21:29] zeljkof, lgtm, please deploy [12:21:52] Urbanecm: ok [12:22:46] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:479723|Enable extendedmover user group at ur.wiki (T211978)]] (duration: 00m 45s) [12:22:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:22:49] T211978: Create page mover group at ur.wiki - https://phabricator.wikimedia.org/T211978 [12:24:56] !log lvs2006: bounce pybal to pick up new service kartotherian-ssl [12:24:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:25:36] Urbanecm: deployed, please test [12:25:41] ack [12:25:57] Urbanecm: I guess 479713 does not need to be tested, right? [12:26:05] yup [12:27:11] (03CR) 10Zfilipin: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479713 (https://phabricator.wikimedia.org/T211795) (owner: 10Tulsi Bhagat) [12:28:13] (03Merged) 10jenkins-bot: Update be.wikisource logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479713 (https://phabricator.wikimedia.org/T211795) (owner: 10Tulsi Bhagat) [12:28:39] (03PS2) 10Arturo Borrero Gonzalez: toolforge: webservicemonitor is now in cron nodes [puppet] - 10https://gerrit.wikimedia.org/r/479737 (https://phabricator.wikimedia.org/T211977) [12:29:18] !log lvs1006: bounce pybal to pick up new service kartotherian-ssl T211970 [12:29:18] !log zfilipin@deploy1001 Synchronized static/images/project-logos/: SWAT: [[gerrit:479713|Update be.wikisource logo (T211795)]] (duration: 00m 45s) [12:29:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:21] T211970: kartotherian TLS support - https://phabricator.wikimedia.org/T211970 [12:29:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:24] T211795: be.wikisource logo issues - https://phabricator.wikimedia.org/T211795 [12:30:33] Urbanecm: 479713 deployed and purged [12:30:36] thx [12:31:17] (03CR) 10Zfilipin: "rebase fixed the problem" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [12:31:29] (03PS4) 10Zfilipin: Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [12:33:00] (03CR) 10Zfilipin: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [12:33:30] (03PS3) 10Arturo Borrero Gonzalez: toolforge: declare sonofgridengime::submit_host from a central point [puppet] - 10https://gerrit.wikimedia.org/r/479736 [12:34:02] (03CR) 10jenkins-bot: Enable 'extendedmover' user group at ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [12:34:06] (03CR) 10jenkins-bot: Update be.wikisource logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479713 (https://phabricator.wikimedia.org/T211795) (owner: 10Tulsi Bhagat) [12:34:33] (03Merged) 10jenkins-bot: Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [12:34:56] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] toolforge: declare sonofgridengime::submit_host from a central point [puppet] - 10https://gerrit.wikimedia.org/r/479736 (owner: 10Arturo Borrero Gonzalez) [12:35:09] (03CR) 10jenkins-bot: Add be.wikisource new 1.5x and 2x logos to wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480004 (https://phabricator.wikimedia.org/T150618) (owner: 10Tulsi Bhagat) [12:35:46] Urbanecm: 480004 at mwdebug [12:35:51] ack [12:36:52] zeljkof, lgtm, please deploy [12:37:10] Urbanecm: ok [12:37:59] !log lvs2003: bounce pybal for new service kartotherian-ssl T211970 [12:38:00] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:480004|Add be.wikisource new 1.5x and 2x logos to wgLogoHD (T150618)]] (duration: 00m 44s) [12:38:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:02] T211970: kartotherian TLS support - https://phabricator.wikimedia.org/T211970 [12:38:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:04] T150618: Provide HD logos for all projects - https://phabricator.wikimedia.org/T150618 [12:40:08] 10Operations, 10Scoring-platform-team, 10Wikimedia-Incident: ORES overload incident, 2017-11-28 - https://phabricator.wikimedia.org/T181538 (10Ladsgroup) [12:40:11] 10Operations, 10Scoring-platform-team (Current), 10User-Ladsgroup, 10Wikimedia-Incident: Investigate redis-cluster or other techniques for making Redis not a single point of failure. - https://phabricator.wikimedia.org/T181559 (10Ladsgroup) 05Open→03Resolved [12:40:13] 10Operations, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842 (10Ladsgroup) 05Open→03Resolved [12:40:15] Urbanecm: deployed, please test and thanks for deploying with #releng ;) [12:40:19] Amir1: I'm done [12:40:30] thanks [12:40:50] akosiaris: For when you have a minute: https://gerrit.wikimedia.org/r/c/operations/puppet/+/479715 [12:41:15] (03CR) 10Ladsgroup: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/473510 (https://phabricator.wikimedia.org/T206777) (owner: 10Reedy) [12:41:24] (03CR) 10Ladsgroup: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477771 (https://phabricator.wikimedia.org/T209820) (owner: 10Ladsgroup) [12:42:28] (03Merged) 10jenkins-bot: Add shnwiki to InterwikiSortOrders.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/473510 (https://phabricator.wikimedia.org/T206777) (owner: 10Reedy) [12:42:56] (03Merged) 10jenkins-bot: Add yue to InterwikiSortOrders.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477771 (https://phabricator.wikimedia.org/T209820) (owner: 10Ladsgroup) [12:43:13] zeljkof, yw, I don't have any other possibility :D [12:43:33] Urbanecm: soon :) [12:43:51] :) [12:44:15] +1 [12:47:44] (03CR) 10jenkins-bot: Add shnwiki to InterwikiSortOrders.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/473510 (https://phabricator.wikimedia.org/T206777) (owner: 10Reedy) [12:47:46] (03CR) 10jenkins-bot: Add yue to InterwikiSortOrders.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477771 (https://phabricator.wikimedia.org/T209820) (owner: 10Ladsgroup) [12:49:03] !log ladsgroup@deploy1001 sync-file aborted: Use the right index for change_tag (T211896) (duration: 00m 02s) [12:49:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:49:06] T211896: Query trying to use the wrong index (change_tag_rev_tag) on change_tag - https://phabricator.wikimedia.org/T211896 [12:49:22] ^ wrong message, aborted [12:50:24] !log ladsgroup@deploy1001 Synchronized wmf-config/InterwikiSortOrders.php: Update InterwikiSortOrders.php (T209820) (duration: 00m 45s) [12:50:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:50:27] T209820: Add Wikidata support to new wikis - https://phabricator.wikimedia.org/T209820 [12:51:14] I'm done [12:51:22] (03CR) 10Alexandros Kosiaris: [C: 03+2] ores: Allow cross origin requests if 'X-Wikimedia-Debug' header is sent [puppet] - 10https://gerrit.wikimedia.org/r/479715 (https://phabricator.wikimedia.org/T211511) (owner: 10Ladsgroup) [12:51:29] (03PS2) 10Alexandros Kosiaris: ores: Allow cross origin requests if 'X-Wikimedia-Debug' header is sent [puppet] - 10https://gerrit.wikimedia.org/r/479715 (https://phabricator.wikimedia.org/T211511) (owner: 10Ladsgroup) [12:51:41] (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] ores: Allow cross origin requests if 'X-Wikimedia-Debug' header is sent [puppet] - 10https://gerrit.wikimedia.org/r/479715 (https://phabricator.wikimedia.org/T211511) (owner: 10Ladsgroup) [12:52:54] !log lvs1016: bounce pybal for new service kartotherian-ssl T211970 [12:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:52:57] T211970: kartotherian TLS support - https://phabricator.wikimedia.org/T211970 [13:02:25] !log ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=fawiki --force-protocol https [13:02:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:04:09] !log ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https [13:04:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:04:20] This might break things [13:04:57] !log ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T209820) [13:04:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:04:59] T209820: Add Wikidata support to new wikis - https://phabricator.wikimedia.org/T209820 [13:06:39] (03PS4) 10ArielGlenn: switch all invocations of scripts in dumps rep to use python3 [puppet] - 10https://gerrit.wikimedia.org/r/479985 (https://phabricator.wikimedia.org/T210989) [13:07:44] 10Puppet, 10Cloud-VPS: Something is wrong with puppet on the jessie bootstrapvz instance - https://phabricator.wikimedia.org/T212119 (10Krenair) [13:11:08] 10Operations, 10Traffic, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:11:13] 10Operations, 10Traffic, 10Maps (Kartotherian), 10Patch-For-Review: kartotherian TLS support - https://phabricator.wikimedia.org/T211970 (10ema) 05Open→03Resolved [13:13:10] jouncebot: now [13:13:10] No deployments scheduled for the next 4 hour(s) and 46 minute(s) [13:14:35] (03CR) 10ArielGlenn: [C: 03+2] convert dump scripts to python3 [dumps] - 10https://gerrit.wikimedia.org/r/478702 (https://phabricator.wikimedia.org/T210989) (owner: 10ArielGlenn) [13:15:09] jouncebot update [13:15:28] 10Operations, 10Traffic, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) I do agree with @Joe, without a proper PKI this is going to be painful. For now however I've added TLS support to kartotherian (T211970) as that's part of cache_upload, which is the cluste... [13:15:31] jouncebot: now [13:15:31] No deployments scheduled for the next 4 hour(s) and 44 minute(s) [13:15:35] jouncebot: update [13:15:42] 10Operations, 10Traffic, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:15:45] why wont you update :/ [13:15:47] !log ariel@deploy1001 Started deploy [dumps/dumps@0393ca7]: switch dumps to python3 [13:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:15:50] !log ariel@deploy1001 Finished deploy [dumps/dumps@0393ca7]: switch dumps to python3 (duration: 00m 03s) [13:15:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:16:05] jouncebot: update [13:16:19] no updates for u [13:16:39] that is the right command though? [13:16:43] im not just going mad [13:16:45] (03PS5) 10ArielGlenn: switch all invocations of scripts in dumps rep to use python3 [puppet] - 10https://gerrit.wikimedia.org/r/479985 (https://phabricator.wikimedia.org/T210989) [13:17:57] uh lesse [13:18:34] addshore: I'm rebulding sites tables atm [13:18:37] htwiki now [13:18:55] Amir1: ack [13:19:03] (03PS1) 10Addshore: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480067 (https://phabricator.wikimedia.org/T201837) [13:19:19] jouncebot: refresh [13:19:20] I refreshed my knowledge about deployments. [13:19:26] jouncebot: now [13:19:26] For the next 1 hour(s) and 40 minute(s): Wikibase / Wikidata (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1300) [13:19:26] Reedy: aaaha, i had it wrong ;) [13:19:27] jouncebot: next [13:19:27] In 4 hour(s) and 40 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1800) [13:19:43] (03PS6) 10ArielGlenn: switch all invocations of scripts in dumps rep to use python3 [puppet] - 10https://gerrit.wikimedia.org/r/479985 (https://phabricator.wikimedia.org/T210989) [13:19:48] heh [13:19:53] i was indeed going mad [13:20:11] Amir1: how long will that take> [13:20:12] I was looking but only with 1/2 eye (sorry), I'm in the middle of likely breaking all my stuff so... [13:20:30] itwiki now [13:20:37] ten minutes-ish [13:20:49] okay, im gonna go and get a coffee then I'll do my thing! [13:21:04] it failed for incubatorwiki yay. I need to re-run it there [13:21:15] (03CR) 10ArielGlenn: [C: 03+2] switch all invocations of scripts in dumps rep to use python3 [puppet] - 10https://gerrit.wikimedia.org/r/479985 (https://phabricator.wikimedia.org/T210989) (owner: 10ArielGlenn) [13:29:19] (03PS2) 10Volans: remote: add more functionalities [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) [13:29:27] !log deleting everything from site_identifiers@incubatorwiki [13:29:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:31:03] !log ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki incubatorwiki --force-protocol https [13:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:31:26] Amir1: done? :D [13:31:32] pswiki now :D [13:31:34] ;) [13:31:43] * addshore will wait for the ping, i didn't even go get my coffee yet [13:34:02] (03PS3) 10Arturo Borrero Gonzalez: toolforge: webservicemonitor is now in cron nodes [puppet] - 10https://gerrit.wikimedia.org/r/479737 (https://phabricator.wikimedia.org/T211977) [13:35:44] sourceswiki failed too [13:36:19] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] toolforge: webservicemonitor is now in cron nodes [puppet] - 10https://gerrit.wikimedia.org/r/479737 (https://phabricator.wikimedia.org/T211977) (owner: 10Arturo Borrero Gonzalez) [13:36:28] rebuilt it manually [13:40:35] 10Operations, 10Thumbor, 10serviceops, 10Performance-Team (Radar), 10User-jijiki: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10jijiki) [13:40:49] 10Operations: Netbox: fill network topology - https://phabricator.wikimedia.org/T205897 (10Volans) @ayounsi do you want to keep this open for tracking things to integrate further although the goal part is done? [13:40:59] !log installing new version of php-excimer on mwdebug2001 - T205059 [13:41:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:41:02] T205059: Excimer: new profiler for PHP - https://phabricator.wikimedia.org/T205059 [13:43:31] addshore: done now [13:44:07] Amir1: sweet [13:44:23] (03CR) 10Addshore: [C: 03+2] wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480067 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [13:45:31] (03Merged) 10jenkins-bot: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480067 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [13:46:16] !log installing new version of php-excimer on mwdebug* - T205059 [13:46:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:46:19] T205059: Excimer: new profiler for PHP - https://phabricator.wikimedia.org/T205059 [13:49:15] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki T201837 (duration: 00m 45s) [13:49:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:49:18] T201837: Use link formatter that uses cache instead of wb_terms for all wikidatawiki items - https://phabricator.wikimedia.org/T201837 [13:49:27] !log Enable GTID on s3 codfw master db2043 - T211973 [13:49:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:49:30] T211973: Check GTID, consistency options and notifications across the fleet - https://phabricator.wikimedia.org/T211973 [13:49:55] (03PS1) 10Volans: sre.hosts: add upgrade and reboot cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480072 (https://phabricator.wikimedia.org/T205886) [13:50:08] (03PS1) 10Addshore: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480073 (https://phabricator.wikimedia.org/T201837) [13:50:38] (03CR) 10jerkins-bot: [V: 04-1] sre.hosts: add upgrade and reboot cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480072 (https://phabricator.wikimedia.org/T205886) (owner: 10Volans) [13:50:41] (03PS3) 10Elukey: mcrouter: temporary remove mc2033 to ease network maintenance [puppet] - 10https://gerrit.wikimedia.org/r/477473 (https://phabricator.wikimedia.org/T210467) [13:51:05] (03PS1) 10Daimona Eaytoy: Rename a global variable in AbuseFilter config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480074 [13:52:12] (03PS2) 10Daimona Eaytoy: Rename a global variable in AbuseFilter config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480074 [13:52:14] (03CR) 10jenkins-bot: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480067 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [13:53:28] (03PS2) 10Volans: sre.hosts: add upgrade and reboot cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480072 (https://phabricator.wikimedia.org/T205886) [13:55:13] (03CR) 10Addshore: [C: 03+2] wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480073 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [13:56:18] (03Merged) 10jenkins-bot: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480073 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [13:58:25] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki T201837 (duration: 00m 45s) [13:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:29] T201837: Use link formatter that uses cache instead of wb_terms for all wikidatawiki items - https://phabricator.wikimedia.org/T201837 [13:59:39] (03PS1) 10Effie Mouzeli: aptrepo: add component/thumbor to stretch-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/480077 (https://phabricator.wikimedia.org/T209886) [13:59:49] 10Operations, 10Prod-Kubernetes, 10serviceops, 10Continuous-Integration-Infrastructure (shipyard), and 2 others: debianize docker-registry 2.7.0-rc0 and upload in stretch-wikimedia - https://phabricator.wikimedia.org/T210071 (10fselles) [13:59:57] 10Operations, 10Prod-Kubernetes, 10serviceops, 10Continuous-Integration-Infrastructure (shipyard), 10Kubernetes: improve docker registry architecture - https://phabricator.wikimedia.org/T209271 (10fselles) [14:01:24] jouncebot: next [14:01:24] In 3 hour(s) and 58 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1800) [14:02:58] (03PS1) 10Marostegui: db-codfw.php: Depool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480078 [14:03:38] addshore: Can I deploy ^ ? [14:03:45] marostegui: go for it [14:03:48] Thanks! [14:04:05] (03CR) 10Marostegui: [C: 03+2] db-codfw.php: Depool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480078 (owner: 10Marostegui) [14:04:59] (03CR) 10jenkins-bot: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480073 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [14:05:05] (03PS1) 10Addshore: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480080 (https://phabricator.wikimedia.org/T201837) [14:05:10] (03Merged) 10jenkins-bot: db-codfw.php: Depool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480078 (owner: 10Marostegui) [14:05:20] (03PS2) 10Addshore: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480080 (https://phabricator.wikimedia.org/T201837) [14:05:26] (03CR) 10jenkins-bot: db-codfw.php: Depool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480078 (owner: 10Marostegui) [14:06:12] marostegui: ping me when I can go again! [14:06:15] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Depool db2038 for mysql and kernel upgrade (duration: 00m 44s) [14:06:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:06:21] addshore: now! :) [14:06:28] !log Stop MySQL on db2038 for mysql and kernel upgrade [14:06:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:06:33] woo! [14:06:53] (03CR) 10Addshore: [C: 03+2] wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480080 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [14:08:05] (03Merged) 10jenkins-bot: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480080 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [14:09:13] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere T201837 (duration: 00m 44s) [14:09:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:20] T201837: Use link formatter that uses cache instead of wb_terms for all wikidatawiki items - https://phabricator.wikimedia.org/T201837 [14:10:50] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere T201837 (duration: 00m 44s) [14:10:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:11:15] (03PS1) 10Andrew Bogott: Horizon: move more projects to eqiad1: [puppet] - 10https://gerrit.wikimedia.org/r/480082 (https://phabricator.wikimedia.org/T204745) [14:13:21] Amir1: FYI i just spotted this [14:13:21] [XBeu1ApAAD8AAHrJB9AAAAAJ] /wiki/Special:EntityData/Q7174.ttl?nocache=1545055956414&flavor=dump ErrorException from line 155 of /srv/mediawiki/php-1.33.0-wmf.8/includes/site/SiteList.php: PHP Notice: Undefined index: shnwiki [14:13:27] not sure if that relates to what you were doing? [14:13:49] it indeed is, it's because of missing cache [14:14:02] it should be fixed by 14:40 [14:14:20] I'm looking for a way to blow the cache now [14:15:01] (03CR) 10Andrew Bogott: [C: 03+2] Horizon: move more projects to eqiad1: [puppet] - 10https://gerrit.wikimedia.org/r/480082 (https://phabricator.wikimedia.org/T204745) (owner: 10Andrew Bogott) [14:15:47] * addshore is seeing the bytes read for mc1022 shoot up, wondering if that is the final change i just made [14:16:15] 10Operations, 10monitoring, 10Patch-For-Review, 10Performance-Team (Radar): Provision >= 50% of statsd/Graphite-only metrics in Prometheus - https://phabricator.wikimedia.org/T205870 (10Peter) Yes, - browsertime is the correct one, the other one we should remove. [14:17:02] 10Operations, 10ORES, 10Security-Team, 10Patch-For-Review, and 2 others: Fetching ORES API from en.wikipedia.org blocked in debug mode - https://phabricator.wikimedia.org/T211511 (10Ladsgroup) 05Open→03Resolved This is done. [14:17:31] looks like it just shot up and down again [14:17:35] (03CR) 10jenkins-bot: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480080 (https://phabricator.wikimedia.org/T201837) (owner: 10Addshore) [14:17:49] 10Operations, 10monitoring, 10Patch-For-Review, 10Performance-Team (Radar): Provision >= 50% of statsd/Graphite-only metrics in Prometheus - https://phabricator.wikimedia.org/T205870 (10fgiunchedi) [14:18:47] (03PS1) 10Ema: ATS: use TLS to access kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/480083 (https://phabricator.wikimedia.org/T207048) [14:22:02] (03CR) 10Ema: [C: 03+2] ATS: use TLS to access kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/480083 (https://phabricator.wikimedia.org/T207048) (owner: 10Ema) [14:23:06] (03PS4) 10Elukey: mcrouter: temporary remove mc2033 to ease network maintenance [puppet] - 10https://gerrit.wikimedia.org/r/477473 (https://phabricator.wikimedia.org/T210467) [14:25:21] (03CR) 10Effie Mouzeli: [C: 03+2] mcrouter: temporary remove mc2033 to ease network maintenance [puppet] - 10https://gerrit.wikimedia.org/r/477473 (https://phabricator.wikimedia.org/T210467) (owner: 10Elukey) [14:28:19] 10Operations, 10Prod-Kubernetes, 10serviceops, 10Continuous-Integration-Infrastructure (shipyard), 10Kubernetes: improve docker registry architecture - https://phabricator.wikimedia.org/T209271 (10fselles) [14:31:57] (03PS1) 10Arturo Borrero Gonzalez: d/changelog: fix format typo [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480084 [14:33:02] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] d/changelog: fix format typo [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480084 (owner: 10Arturo Borrero Gonzalez) [14:33:18] !log Upgrade mysql and kernel on db2093 (tendril sby host on codfw) [14:33:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:34:38] (03CR) 10Mathew.onipe: [C: 03+1] ATS: use TLS to access kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/480083 (https://phabricator.wikimedia.org/T207048) (owner: 10Ema) [14:39:10] (03PS2) 10Herron: add load balanced service kibana.svc.codfw.wmnet (eqiad backup) [puppet] - 10https://gerrit.wikimedia.org/r/479541 (https://phabricator.wikimedia.org/T205850) [14:39:57] (03PS1) 10Arturo Borrero Gonzalez: tools-manifest: bump debhelper version to 11 [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480085 (https://phabricator.wikimedia.org/T211977) [14:39:59] (03PS1) 10Arturo Borrero Gonzalez: tools-manifest: add systemd support [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480086 (https://phabricator.wikimedia.org/T211977) [14:40:01] (03PS1) 10Arturo Borrero Gonzalez: d/changelog: generate entry for 0.13 stretch [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480087 [14:40:26] (03CR) 10jerkins-bot: [V: 04-1] tools-manifest: add systemd support [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480086 (https://phabricator.wikimedia.org/T211977) (owner: 10Arturo Borrero Gonzalez) [14:40:28] (03CR) 10jerkins-bot: [V: 04-1] tools-manifest: bump debhelper version to 11 [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480085 (https://phabricator.wikimedia.org/T211977) (owner: 10Arturo Borrero Gonzalez) [14:42:51] (03PS3) 10Herron: add load balanced service kibana.svc.codfw.wmnet (eqiad backup) [puppet] - 10https://gerrit.wikimedia.org/r/479541 (https://phabricator.wikimedia.org/T205850) [14:48:01] (03CR) 10Herron: [C: 03+2] add load balanced service kibana.svc.codfw.wmnet (eqiad backup) [puppet] - 10https://gerrit.wikimedia.org/r/479541 (https://phabricator.wikimedia.org/T205850) (owner: 10Herron) [14:55:54] 10Operations, 10Core Platform Team, 10MediaWiki-Cache, 10Performance-Team, 10serviceops: Use a multi-dc aware store for `wgMainStash` if needed. - https://phabricator.wikimedia.org/T212129 (10Joe) [15:00:41] (03CR) 10Ema: [C: 03+1] "Looks good, just two minor comments." (032 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [15:02:48] (03CR) 10Alexandros Kosiaris: [C: 03+1] "We don't really use /var/lib/docker and /var/lib/kubelet in node exporter metrics (we rather rely on kubelet's cadvisor prometheus metrics" [puppet] - 10https://gerrit.wikimedia.org/r/479424 (https://phabricator.wikimedia.org/T211810) (owner: 10GTirloni) [15:04:57] 10Operations, 10Operations-Software-Development, 10serviceops, 10User-Joe, 10User-jijiki: Convert makevm to spicerack cookbook - https://phabricator.wikimedia.org/T203963 (10akosiaris) [15:06:40] (03CR) 10Addshore: Specify $wgWBRepoSettings['conceptBaseUri'] (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477521 (owner: 10Lucas Werkmeister (WMDE)) [15:08:42] (03CR) 10Arturo Borrero Gonzalez: "> Main test build failed." [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480085 (https://phabricator.wikimedia.org/T211977) (owner: 10Arturo Borrero Gonzalez) [15:08:45] (03CR) 10Ema: remote: add more functionalities (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [15:18:23] PROBLEM - Host kibana.svc.codfw.wmnet is DOWN: PING CRITICAL - Packet loss = 100% [15:18:47] <_joe_> uh [15:18:49] ^ that’s me new service [15:18:51] herron: is that you? [15:18:53] ack [15:18:54] <_joe_> herron: ok [15:18:55] οκ [15:18:57] ack [15:19:15] <_joe_> I was about to go brew coffee and I thought it was a sign from the Gods of Production [15:19:28] is this the usual problem of it being impossible to downtime a newly-created service? [15:19:45] <_joe_> cdanis: slightly different, and easily solved with a cookbook [15:19:55] <_joe_> this is the "add a new lvs endpoint" race condition [15:20:45] kind of. This is actually indeed possible to avoid. [15:20:54] <_joe_> akosiaris: no it's not [15:20:59] nope, it is [15:21:10] look at eventstreams in lvs::configuration [15:21:13] <_joe_> akosiaris: ok look I added two new services to lvs last week [15:21:14] like 1069 [15:21:19] line 1069 [15:21:23] <_joe_> and neither paged [15:21:34] critical: false :-). You can avoid the pages while setting it up [15:21:47] <_joe_> because I coordinated the change, including disabling puppet on icinga1001 [15:21:51] <_joe_> but yes, that's the alternative [15:22:00] <_joe_> I was talking about doing it in one commit [15:22:07] <_joe_> which is ok in many cases [15:22:30] <_joe_> now doing all the coordination work by hand is too much hassle normally [15:22:45] I have to say that doing it in 2 commits and doing it without having to orchestrate it sounds more painful [15:22:49] <_joe_> so in those cases, yes, 2 commits [15:23:00] <_joe_> akosiaris: yeah either 2 commits or a cookbook [15:23:10] <_joe_> anyways, gotta go brew that coffee [15:23:11] <_joe_> :P [15:25:21] (03PS2) 10Herron: add forward/reverse records for kibana.svc.codfw.wmnet [dns] - 10https://gerrit.wikimedia.org/r/479539 [15:25:57] (03PS2) 10Volans: icinga: add execution of Icinga commands [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) [15:25:59] (03PS3) 10Volans: remote: add more functionalities [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) [15:26:10] (03CR) 10Volans: "replies inline" (032 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [15:26:14] (03CR) 10Herron: [C: 03+2] add forward/reverse records for kibana.svc.codfw.wmnet [dns] - 10https://gerrit.wikimedia.org/r/479539 (owner: 10Herron) [15:26:38] (03CR) 10Volans: "Reply inline" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [15:38:18] (03PS2) 10Mforns: Update analytics eventlogging_to_druid_job.pp to mirror changes in scala job [puppet] - 10https://gerrit.wikimedia.org/r/479847 (https://phabricator.wikimedia.org/T210099) [15:41:20] (03CR) 10Ema: remote: add more functionalities (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [15:42:04] (03CR) 10Ema: [C: 03+1] icinga: add execution of Icinga commands (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [15:44:20] (03PS1) 10Herron: logstash: add lvs::realserver to codfw logstash collector nodes [puppet] - 10https://gerrit.wikimedia.org/r/480096 (https://phabricator.wikimedia.org/T205850) [15:49:50] (03CR) 10Herron: [C: 03+2] logstash: add lvs::realserver to codfw logstash collector nodes [puppet] - 10https://gerrit.wikimedia.org/r/480096 (https://phabricator.wikimedia.org/T205850) (owner: 10Herron) [15:56:51] (03CR) 10Mforns: [C: 04-1] "This should be merged together with the next refinery-source deployment." [puppet] - 10https://gerrit.wikimedia.org/r/479847 (https://phabricator.wikimedia.org/T210099) (owner: 10Mforns) [16:07:16] !log mobrovac@deploy1001 Started restart [electron-render/deploy@94d27d7]: Electron stuck [16:07:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:32] !log anomie@mwmaint1002 Running migrateActors.php on test wikis and mediawikiwiki for T188327 [16:07:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:35] T188327: Deploy refactored actor storage - https://phabricator.wikimedia.org/T188327 [16:13:21] !log Aborted migrateActors.php run, queries were too slow. [16:13:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:16:11] (03PS3) 10CRusnov: Change all reports to log only errors except for a summary count [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/479583 (https://phabricator.wikimedia.org/T205899) [16:16:11] PROBLEM - BGP status on cr2-esams is CRITICAL: BGP CRITICAL - AS6939/IPv6: Active https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [16:17:57] 10Operations, 10Operations-Software-Development, 10Goal: Expand Netbox usage - Q2 2018-19 Goal - https://phabricator.wikimedia.org/T205868 (10ayounsi) [16:17:59] 10Operations: Netbox: fill network topology - https://phabricator.wikimedia.org/T205897 (10ayounsi) 05Open→03Resolved It's fine to close it and open a new one when we know how to proceed to import need data. [16:25:47] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [16:26:21] (03PS1) 10Herron: add codfw to conftool kibana service datacenter list [puppet] - 10https://gerrit.wikimedia.org/r/480102 (https://phabricator.wikimedia.org/T205850) [16:29:06] (03CR) 10Herron: [C: 03+2] add codfw to conftool kibana service datacenter list [puppet] - 10https://gerrit.wikimedia.org/r/480102 (https://phabricator.wikimedia.org/T205850) (owner: 10Herron) [16:31:51] 10Operations, 10Developer-Advocacy, 10Gerrit, 10serviceops: Remove port 29418 from cloning process - https://phabricator.wikimedia.org/T37611 (10Dzahn) [16:32:20] 10Operations, 10serviceops, 10vm-requests, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) [16:32:52] (03PS1) 10Ema: sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) [16:32:57] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [16:33:01] (03PS1) 10Alexandros Kosiaris: Add eevans, mobrovac, ppchelko to proton-admins [puppet] - 10https://gerrit.wikimedia.org/r/480104 [16:33:05] PROBLEM - restbase endpoints health on restbase-dev1005 is CRITICAL: /en.wikipedia.org/v1/data/citation/{format}/{query} (Get citation for Darth Vader) timed out before a response was received [16:33:15] !log installing php5 security updates on jessie [16:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:33:20] 10Operations, 10serviceops: docker-registry.wikimedia.org caches images missing instead of revalidating - https://phabricator.wikimedia.org/T211719 (10Dzahn) [16:33:30] (03CR) 10jerkins-bot: [V: 04-1] sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) (owner: 10Ema) [16:34:11] RECOVERY - restbase endpoints health on restbase-dev1005 is OK: All endpoints are healthy [16:35:12] (03PS1) 10Kosta Harlan: Production configuration for GrowthExperiments Help Panel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) [16:35:56] 10Operations, 10Phabricator, 10serviceops, 10Patch-For-Review, and 4 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Dzahn) [16:36:17] (03CR) 10jerkins-bot: [V: 04-1] Production configuration for GrowthExperiments Help Panel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) (owner: 10Kosta Harlan) [16:37:16] (03PS2) 10Kosta Harlan: Production configuration for GrowthExperiments Help Panel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) [16:37:22] (03PS2) 10Ema: sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) [16:38:01] (03CR) 10jerkins-bot: [V: 04-1] sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) (owner: 10Ema) [16:38:40] 10Operations, 10Icinga, 10monitoring: Icinga passive checks go awal and downtime stops working - https://phabricator.wikimedia.org/T196336 (10Dzahn) 05Open→03Resolved a:03Dzahn Based on further comments by both Jaime and Riccardo that this could be closed and reopened if needed.. and that we have a new... [16:40:11] PROBLEM - citoid endpoints health on scb1004 is CRITICAL: /api (Scrapes sample page) timed out before a response was received [16:41:23] RECOVERY - citoid endpoints health on scb1004 is OK: All endpoints are healthy [16:41:29] !log installing libarchive security updates on jessie [16:41:30] (03CR) 10Sbisson: Production configuration for GrowthExperiments Help Panel (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) (owner: 10Kosta Harlan) [16:41:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:41:37] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Scrapes sample page) timed out before a response was received [16:42:55] (03CR) 10Kosta Harlan: Production configuration for GrowthExperiments Help Panel (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) (owner: 10Kosta Harlan) [16:43:20] (03CR) 10Urbanecm: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) (owner: 10Kosta Harlan) [16:44:03] PROBLEM - citoid endpoints health on scb1002 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [16:44:49] (03PS3) 10Ema: sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) [16:45:09] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [16:45:15] PROBLEM - citoid endpoints health on scb1003 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [16:45:30] (03CR) 10jerkins-bot: [V: 04-1] sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) (owner: 10Ema) [16:46:21] RECOVERY - citoid endpoints health on scb1002 is OK: All endpoints are healthy [16:46:25] (03PS4) 10Ema: sre.hosts: add varnish upgrade cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) [16:47:35] RECOVERY - citoid endpoints health on scb1003 is OK: All endpoints are healthy [16:51:32] (03CR) 10Mark Bergsma: [C: 03+2] Extend testConfigServerRemoval test case. [debs/pybal] - 10https://gerrit.wikimedia.org/r/447770 (owner: 10Mark Bergsma) [16:52:33] (03Merged) 10jenkins-bot: Extend testConfigServerRemoval test case. [debs/pybal] - 10https://gerrit.wikimedia.org/r/447770 (owner: 10Mark Bergsma) [16:52:35] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [16:53:41] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [16:54:29] (03PS3) 10Kosta Harlan: Production configuration for GrowthExperiments Help Panel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) [16:57:17] (03CR) 10Kosta Harlan: Production configuration for GrowthExperiments Help Panel (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480106 (https://phabricator.wikimedia.org/T211991) (owner: 10Kosta Harlan) [16:58:34] (03CR) 10Dzahn: [C: 03+1] Add eevans, mobrovac, ppchelko to proton-admins [puppet] - 10https://gerrit.wikimedia.org/r/480104 (owner: 10Alexandros Kosiaris) [16:59:11] 10Operations, 10ops-eqiad, 10media-storage, 10User-fgiunchedi: rack/setup/install ms-be10[44-50].eqiad.wmnet - https://phabricator.wikimedia.org/T209618 (10RobH) TL;DR / IRC Summary Update: The new systems are not distributed as evenly across all 4 rows as needed for our existing horizontal redundancy.... [17:04:38] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/480077 (https://phabricator.wikimedia.org/T209886) (owner: 10Effie Mouzeli) [17:13:09] (03PS3) 10Michael Große: Perform even more PHP constraint checks before falling back [mediawiki-config] - 10https://gerrit.wikimedia.org/r/478630 (https://phabricator.wikimedia.org/T209504) [17:14:45] PROBLEM - Host ms-be2047.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:18:35] (03PS1) 10Alexandros Kosiaris: Add zotero specific network policy exceptions to default one [puppet] - 10https://gerrit.wikimedia.org/r/480114 [17:19:54] 10Operations, 10Performance-Team, 10Graphite: Graphite generates a lot of 502 in Grafana - https://phabricator.wikimedia.org/T211747 (10CDanis) We could try changing the Grafana datasource back to 'direct' (now called 'Browser' in 5.x) -- it was flipped to server aka proxy just because of CORS requirements w... [17:20:17] 10Operations, 10Performance-Team, 10monitoring, 10Graphite: Graphite generates a lot of 502 in Grafana - https://phabricator.wikimedia.org/T211747 (10CDanis) [17:21:17] PROBLEM - Host ms-be1048 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:43] PROBLEM - Host ms-be1048.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:24:43] cmjohnson1: are both ms-be1048 and ms-be1047 being worked on? [17:25:31] (03CR) 10Fsero: [C: 03+2] Add zotero specific network policy exceptions to default one [puppet] - 10https://gerrit.wikimedia.org/r/480114 (owner: 10Alexandros Kosiaris) [17:25:35] only 1048 [17:26:05] the other shows mgmt down for ms-be2047, that is papaul [17:26:46] lol, I'm blind:) sorry [17:27:06] (03PS2) 10Alexandros Kosiaris: Add zotero specific network policy exceptions to default one [puppet] - 10https://gerrit.wikimedia.org/r/480114 [17:27:09] (03CR) 10Alexandros Kosiaris: [V: 03+2] Add zotero specific network policy exceptions to default one [puppet] - 10https://gerrit.wikimedia.org/r/480114 (owner: 10Alexandros Kosiaris) [17:28:16] 10Operations, 10ops-codfw, 10User-fgiunchedi: ms-be2047 spontaneous reboots - https://phabricator.wikimedia.org/T209921 (10Papaul) CPU 1 has been replaced. I clear also the log. The system is back up and I will be monitoring it once again. [17:31:01] RECOVERY - Host ms-be2047.mgmt is UP: PING OK - Packet loss = 0%, RTA = 36.96 ms [17:33:41] 10Operations, 10ops-codfw, 10netops: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10Papaul) fpc2-fpc8 connection xe-2/0/41 and xe-2/0/42 fpc7-fpc8 connection xe-7/0/43 and xe-7/0/44 [17:34:01] 10Operations, 10ops-codfw, 10netops: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10Papaul) [17:34:29] RECOVERY - Host ms-be1048.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.10 ms [17:38:02] (03PS2) 10Alexandros Kosiaris: Add eevans, mobrovac, ppchelko to proton-admins [puppet] - 10https://gerrit.wikimedia.org/r/480104 [17:38:10] (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Add eevans, mobrovac, ppchelko to proton-admins [puppet] - 10https://gerrit.wikimedia.org/r/480104 (owner: 10Alexandros Kosiaris) [17:38:55] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review: codfw row B recable and add QFX - https://phabricator.wikimedia.org/T210456 (10Papaul) [17:39:42] (03CR) 10BBlack: New zone generator gen-zones.py (035 comments) [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [17:39:56] 10Operations, 10ops-eqiad, 10media-storage, 10User-fgiunchedi: rack/setup/install ms-be10[44-50].eqiad.wmnet - https://phabricator.wikimedia.org/T209618 (10Cmjohnson) @robh the server has been moved to d2/u31. [x] update netbox [x] remove from asw2-b-eqiad [x] update asw2-d-eqiad [17:39:58] 10Operations, 10ops-codfw, 10netops: upgrade all codfw switch stacks to include additional 10G switch per row - https://phabricator.wikimedia.org/T196489 (10Papaul) [17:40:03] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review: codfw row B recable and add QFX - https://phabricator.wikimedia.org/T210456 (10Papaul) 05Open→03Resolved Complete [17:40:07] (03PS2) 10BBlack: New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 [17:40:18] (03CR) 10jerkins-bot: [V: 04-1] New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [17:43:33] (03PS3) 10BBlack: New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 [17:43:46] (03CR) 10jerkins-bot: [V: 04-1] New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [17:45:01] 10Operations, 10Operations-Software-Development: Introduce Python code formatters usage - https://phabricator.wikimedia.org/T211750 (10GTirloni) I don't have much to add besides saying I've been using [[ https://github.com/ambv/black | Black ]] (with VSCode) for all my new code here. I'd vote for it to be adop... [17:50:13] (03PS1) 10Alexandros Kosiaris: Introduce blubberoid LVS IP endpoints [dns] - 10https://gerrit.wikimedia.org/r/480123 (https://phabricator.wikimedia.org/T205919) [17:53:14] 10Operations, 10Operations-Software-Development, 10Goal: Expand Spicerack library and SRE Cookbooks - Q2 2018-19 Goal - https://phabricator.wikimedia.org/T205867 (10Volans) [17:54:37] (03CR) 10Volans: [C: 03+1] "LGTM" [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/479583 (https://phabricator.wikimedia.org/T205899) (owner: 10CRusnov) [17:59:58] PROBLEM - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is CRITICAL: 137.3 ge 130 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1panelId=2fullscreen [18:00:04] gehel and onimisionipe: Dear deployers, time to do the Wikidata Query Service weekly deploy deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1800). [18:00:25] here here [18:01:08] jouncebot’s jokes are the best [18:02:22] (03CR) 10Volans: [C: 03+1] "I didn't try it, but looks ok to me." (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [18:04:21] (03CR) 10Volans: [C: 04-1] "Nice, thanks for the help. Few comments inline, some are limitations of the library, nice to find real use cases to improve the current li" (037 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/480103 (https://phabricator.wikimedia.org/T205886) (owner: 10Ema) [18:04:47] !log herron@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2004.codfw.wmnet [18:04:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:01] (03CR) 10Volans: [C: 03+2] icinga: add execution of Icinga commands [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [18:07:15] !log akosiaris@deploy1001 scap-helm blubberoid install --name staging --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production stable/blubberoid [namespace: blubberoid, clusters: staging] [18:07:16] !log akosiaris@deploy1001 scap-helm blubberoid cluster staging completed [18:07:16] !log akosiaris@deploy1001 scap-helm blubberoid finished [18:07:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:07:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:07:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:07:43] (03Merged) 10jenkins-bot: icinga: add execution of Icinga commands [software/spicerack] - 10https://gerrit.wikimedia.org/r/480063 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [18:10:05] !log onimisionipe@deploy1001 Started deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater [18:10:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:13:52] (03CR) 10Volans: remote: add more functionalities (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/480064 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [18:14:46] (03PS1) 10Jdlrobson: Page issues treatment enabled on all wikis except enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480127 (https://phabricator.wikimedia.org/T210553) [18:19:07] PROBLEM - recommendation_api endpoints health on scb2005 is CRITICAL: /{domain}/v1/translation/articles/{source}{/seed} (normal source and target with seed) is CRITICAL: Test normal source and target with seed returned the unexpected status 502 (expecting: 200) [18:20:09] RECOVERY - recommendation_api endpoints health on scb2005 is OK: All endpoints are healthy [18:21:50] !log installing libgd2 security updates [18:21:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:21:57] !log onimisionipe@deploy1001 Finished deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater (duration: 11m 52s) [18:21:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:22:09] (03PS1) 10Herron: add codfw site to kibana lvs service configuration [puppet] - 10https://gerrit.wikimedia.org/r/480131 (https://phabricator.wikimedia.org/T205850) [18:22:47] (03CR) 10Ema: [C: 03+1] add codfw site to kibana lvs service configuration [puppet] - 10https://gerrit.wikimedia.org/r/480131 (https://phabricator.wikimedia.org/T205850) (owner: 10Herron) [18:23:13] (03CR) 10Herron: [C: 03+2] add codfw site to kibana lvs service configuration [puppet] - 10https://gerrit.wikimedia.org/r/480131 (https://phabricator.wikimedia.org/T205850) (owner: 10Herron) [18:23:40] !log akosiaris@deploy1001 scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad,codfw] [18:23:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:24:12] (03PS2) 10GTirloni: prometheus-node-exporter: Ignore Docker/Kubelet mount points [puppet] - 10https://gerrit.wikimedia.org/r/479424 (https://phabricator.wikimedia.org/T211810) [18:24:21] !log akosiaris@deploy1001 scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad] [18:24:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:24:48] (03PS1) 10Arturo Borrero Gonzalez: tools-manifest: bump debhelper version to 11 [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480132 (https://phabricator.wikimedia.org/T211977) [18:25:02] !log akosiaris@deploy1001 scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad] [18:25:03] !log akosiaris@deploy1001 scap-helm blubberoid cluster eqiad completed [18:25:03] !log akosiaris@deploy1001 scap-helm blubberoid finished [18:25:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:13] (03CR) 10GTirloni: [C: 03+2] prometheus-node-exporter: Ignore Docker/Kubelet mount points [puppet] - 10https://gerrit.wikimedia.org/r/479424 (https://phabricator.wikimedia.org/T211810) (owner: 10GTirloni) [18:26:16] !log akosiaris@deploy1001 scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.2.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: codfw] [18:26:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:26:18] !log akosiaris@deploy1001 scap-helm blubberoid cluster codfw completed [18:26:18] !log akosiaris@deploy1001 scap-helm blubberoid finished [18:26:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:26:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:27:06] (03PS1) 10Arturo Borrero Gonzalez: tools-manifest: bump debhelper version to 11 [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/480134 (https://phabricator.wikimedia.org/T211977) [18:28:25] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10Papaul) connected to scs-c1-codfw on port 48 [18:28:39] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10Papaul) [18:28:56] (03PS1) 10Arturo Borrero Gonzalez: tools-manifest: bump debhelper version to 11 [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480135 (https://phabricator.wikimedia.org/T211977) [18:29:37] !log prometheus-node-exporter: Ignore Docker/Kubelet mount points T211810 [18:29:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:29:40] T211810: tools-workers: prometheus-node-exporter `Error on statfs() system call for... permission denied` - https://phabricator.wikimedia.org/T211810 [18:30:15] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] tools-manifest: bump debhelper version to 11 [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480135 (https://phabricator.wikimedia.org/T211977) (owner: 10Arturo Borrero Gonzalez) [18:32:31] !log mobrovac@deploy1001 Started deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - T210793 [18:32:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:32:34] T210793: Verify that PDFs generated by Proton service are correct - https://phabricator.wikimedia.org/T210793 [18:32:43] !log performing manual puppet run and manually restarting pybal on lvs2006 to add kibana service T205850 [18:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:32:46] T205850: Procure and provision Logging pipeline hardware in multiple datacenters - https://phabricator.wikimedia.org/T205850 [18:32:58] (03PS1) 10Arturo Borrero Gonzalez: tox.ini: use python3 instead of python3.4 [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480138 [18:33:03] 10Operations, 10serviceops, 10vm-requests, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) a:03Dzahn [18:33:09] 10Operations, 10serviceops, 10vm-requests, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) p:05Triage→03Normal [18:34:03] !log mobrovac@deploy1001 Finished deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - T210793 (duration: 01m 32s) [18:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:34:40] !log herron@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2005.codfw.wmnet [18:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:34:44] !log herron@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2006.codfw.wmnet [18:34:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:35:01] (03CR) 10Bstorm: [C: 03+1] tox.ini: use python3 instead of python3.4 [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480138 (owner: 10Arturo Borrero Gonzalez) [18:35:34] (03CR) 10Arturo Borrero Gonzalez: [C: 04-1] "I like the change in the logic for logging. But please, split the patch into logical changes (linter warnings and logging logic)." [puppet] - 10https://gerrit.wikimedia.org/r/479409 (https://phabricator.wikimedia.org/T211817) (owner: 10GTirloni) [18:35:41] (03CR) 10Bstorm: [C: 03+1] "I'd just include it in https://gerrit.wikimedia.org/r/c/operations/software/tools-manifest/+/480086/, but that doesn't mean this is the wr" [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480138 (owner: 10Arturo Borrero Gonzalez) [18:36:22] RECOVERY - Host kibana.svc.codfw.wmnet is UP: PING OK - Packet loss = 0%, RTA = 36.15 ms [18:36:25] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] tox.ini: use python3 instead of python3.4 [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480138 (owner: 10Arturo Borrero Gonzalez) [18:37:07] (03PS1) 10ArielGlenn: use lbzp2 in wikidata rdf weeklies [puppet] - 10https://gerrit.wikimedia.org/r/480140 (https://phabricator.wikimedia.org/T206535) [18:37:37] I didn't notice the down page, only the recovery one [18:37:40] woops [18:38:34] (03PS1) 10Arturo Borrero Gonzalez: tools-manifest: add systemd support [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480141 (https://phabricator.wikimedia.org/T211977) [18:39:26] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] tools-manifest: add systemd support [software/tools-manifest] (stretch) - 10https://gerrit.wikimedia.org/r/480141 (https://phabricator.wikimedia.org/T211977) (owner: 10Arturo Borrero Gonzalez) [18:42:40] !log manually restarting pybal on lvs2003 to add kibana service T205850 [18:42:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:42:43] T205850: Procure and provision Logging pipeline hardware in multiple datacenters - https://phabricator.wikimedia.org/T205850 [18:43:29] (03PS1) 10Arturo Borrero Gonzalez: Edit Project Config [software/tools-manifest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/480143 [18:45:32] (03PS2) 10Arturo Borrero Gonzalez: Edit Project Config [software/tools-manifest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/480143 [18:46:19] (03CR) 10Arturo Borrero Gonzalez: [V: 03+2 C: 03+2] Edit Project Config [software/tools-manifest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/480143 (owner: 10Arturo Borrero Gonzalez) [18:49:20] (03CR) 10CRusnov: [V: 03+2 C: 03+2] "merging" [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/479583 (https://phabricator.wikimedia.org/T205899) (owner: 10CRusnov) [18:49:55] (03PS1) 10RobH: updating ms-be1048 production dns [dns] - 10https://gerrit.wikimedia.org/r/480145 (https://phabricator.wikimedia.org/T209618) [18:51:10] (03PS2) 10RobH: updating ms-be1048 production dns [dns] - 10https://gerrit.wikimedia.org/r/480145 (https://phabricator.wikimedia.org/T209618) [18:53:19] (03PS2) 10ArielGlenn: use lbzip2 in wikidata rdf weeklies [puppet] - 10https://gerrit.wikimedia.org/r/480140 (https://phabricator.wikimedia.org/T206535) [18:53:55] (03PS2) 10GTirloni: nfs-mount-manager: Reduce verbosity and fix linter warnings [puppet] - 10https://gerrit.wikimedia.org/r/479409 (https://phabricator.wikimedia.org/T211817) [18:54:08] (03CR) 10RobH: [C: 03+2] updating ms-be1048 production dns [dns] - 10https://gerrit.wikimedia.org/r/480145 (https://phabricator.wikimedia.org/T209618) (owner: 10RobH) [18:57:39] (03PS3) 10GTirloni: nfs-mount-manager: Reduce verbosity [puppet] - 10https://gerrit.wikimedia.org/r/479409 (https://phabricator.wikimedia.org/T211817) [18:57:43] !log ppchelko@deploy1001 Started deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001 [18:57:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:58:34] !log ppchelko@deploy1001 Finished deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001 (duration: 00m 51s) [18:58:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:59:17] 10Operations, 10ops-codfw, 10netops: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10ayounsi) [19:00:04] Deploy window Morning SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T1900) [19:00:04] Jdlrobson: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [19:00:21] 10Operations, 10ops-codfw, 10netops: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10ayounsi) Some more shuffling around to facilitate DCops work (as row A has more machines), scheduling row A Tomorrow and row D Wednesday. [19:00:56] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10ayounsi) [19:01:03] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10ayounsi) Some more shuffling around to facilitate DCops work (as row A has more machines), scheduling row A Tomorrow and row D Wednesday. [19:01:09] 10Operations, 10ops-eqiad, 10media-storage, 10Patch-For-Review, 10User-fgiunchedi: rack/setup/install ms-be10[44-50].eqiad.wmnet - https://phabricator.wikimedia.org/T209618 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` ['ms-be1048.eqiad.wmnet']... [19:03:33] here \o [19:04:35] (03PS4) 10GTirloni: nfs-mount-manager: Reduce verbosity [puppet] - 10https://gerrit.wikimedia.org/r/479409 (https://phabricator.wikimedia.org/T211817) [19:04:58] (03PS1) 10Dzahn: introduce doc1001.eqiad.wmnet, assign 10.64.0.142 [dns] - 10https://gerrit.wikimedia.org/r/480151 (https://phabricator.wikimedia.org/T211974) [19:06:45] is anyone available to swat today? [19:12:28] (03PS5) 10GTirloni: nfs-mount-manager: Reduce verbosity [puppet] - 10https://gerrit.wikimedia.org/r/479409 (https://phabricator.wikimedia.org/T211817) [19:12:53] jdlrobson: not sure [19:13:37] (03CR) 10GTirloni: [C: 03+2] nfs-mount-manager: Reduce verbosity [puppet] - 10https://gerrit.wikimedia.org/r/479409 (https://phabricator.wikimedia.org/T211817) (owner: 10GTirloni) [19:13:46] I'll ask around [19:13:59] jdlrobson, I'll do it [19:14:12] :) [19:14:13] Hauskatze, no need, I'll swat [19:14:20] Okay [19:15:05] jdlrobson - give me minute to set up env [19:17:01] (03CR) 10Cwhite: hiera: add alerting_host cluster definition (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/479843 (https://phabricator.wikimedia.org/T210486) (owner: 10Cwhite) [19:17:31] merging https://gerrit.wikimedia.org/r/#/c/480127/ [19:18:28] (03CR) 10Pmiazga: [C: 03+2] Page issues treatment enabled on all wikis except enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480127 (https://phabricator.wikimedia.org/T210553) (owner: 10Jdlrobson) [19:19:33] (03Merged) 10jenkins-bot: Page issues treatment enabled on all wikis except enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480127 (https://phabricator.wikimedia.org/T210553) (owner: 10Jdlrobson) [19:19:35] (03CR) 10GTirloni: "Thanks for the review!" [puppet] - 10https://gerrit.wikimedia.org/r/479424 (https://phabricator.wikimedia.org/T211810) (owner: 10GTirloni) [19:19:55] 10Operations, 10Scap: Canaries canaries canaries - https://phabricator.wikimedia.org/T210143 (10jijiki) [19:22:29] jdlrobson, - it's on mwdebug1002 [19:23:12] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10Papaul) [19:23:38] raynor: I see page issues new treatment in stable on https://ru.m.wikipedia.org/wiki/%D0%9B%D0%B0%D0%B2%D1%80%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9,_%D0%9D%D0%B8%D0%BA%D0%BE%D0%BB%D0%B0%D0%B9_%D0%90%D0%BB%D0%B5%D0%BA%D1%81%D0%B5%D0%B5%D0%B2%D0%B8%D1%87?mobileaction=stable but not https://en.m.wikipedia.org/wiki/Pharmacovigilance?mobileaction=stable [19:23:43] So please sync [19:23:45] working as expected [19:24:50] on it [19:27:03] (03CR) 10jenkins-bot: Page issues treatment enabled on all wikis except enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/480127 (https://phabricator.wikimedia.org/T210553) (owner: 10Jdlrobson) [19:27:08] (03PS4) 10Cwhite: profile: enable statsd_exporter and add matching rules to logstash::collector [puppet] - 10https://gerrit.wikimedia.org/r/479353 (https://phabricator.wikimedia.org/T205870) [19:27:21] (03CR) 10Cwhite: profile: enable statsd_exporter and add matching rules to logstash::collector (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/479353 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [19:27:58] !log pmiazga@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:480127 | Page issues treatment enabled on all wikis except enwiki(T210553)]] (duration: 00m 45s) [19:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:28:00] jdlrobson, it's on production \o/ [19:28:01] T210553: Deploy page issues to all wikipedias (except enwiki) - https://phabricator.wikimedia.org/T210553 [19:29:12] i know that task has noting to do with actual issues with deployment or paging but it had all the right words to make my brain have to re-read it and click link. [19:29:12] !log Morning SWAT finished [19:29:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:31:24] (03PS1) 10Andrew Bogott: dns-floating-ip-updater: handle addresses from Neutron [puppet] - 10https://gerrit.wikimedia.org/r/480158 (https://phabricator.wikimedia.org/T212143) [19:32:45] !log ppchelko@deploy1001 Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001 [19:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:03] !log ppchelko@deploy1001 Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001 (duration: 00m 18s) [19:33:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:09] (03PS9) 10Cwhite: ci: define statsd prometheus exporter mappings [puppet] - 10https://gerrit.wikimedia.org/r/479139 (https://phabricator.wikimedia.org/T205870) [19:33:58] (03CR) 10Cwhite: "That's embarrassing. Fixed." [puppet] - 10https://gerrit.wikimedia.org/r/479139 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [19:34:34] (03CR) 10Dzahn: [C: 03+2] introduce doc1001.eqiad.wmnet, assign 10.64.0.142 [dns] - 10https://gerrit.wikimedia.org/r/480151 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [19:36:37] !log ppchelko@deploy1001 Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts [19:36:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:36:51] (03PS1) 10MaxSem: php72: add RSVG [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/480159 (https://phabricator.wikimedia.org/T151656) [19:37:03] (03CR) 10Cwhite: [C: 03+1] Remove Diamond from further roles [puppet] - 10https://gerrit.wikimedia.org/r/480032 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [19:38:18] 10Operations, 10ops-codfw, 10netops, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10ayounsi) [19:39:27] (03PS1) 10Bstorm: sonofgridengine: fixing the master service so it operates best for autostart [puppet] - 10https://gerrit.wikimedia.org/r/480160 (https://phabricator.wikimedia.org/T211258) [19:40:37] (03CR) 10Bstorm: [C: 03+2] sonofgridengine: fixing the master service so it operates best for autostart [puppet] - 10https://gerrit.wikimedia.org/r/480160 (https://phabricator.wikimedia.org/T211258) (owner: 10Bstorm) [19:40:47] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] sonofgridengine: fixing the master service so it operates best for autostart [puppet] - 10https://gerrit.wikimedia.org/r/480160 (https://phabricator.wikimedia.org/T211258) (owner: 10Bstorm) [19:41:02] (03CR) 10Alex Monk: [C: 03+1] "weird I thought we did this" [puppet] - 10https://gerrit.wikimedia.org/r/480158 (https://phabricator.wikimedia.org/T212143) (owner: 10Andrew Bogott) [19:41:24] (03CR) 10Andrew Bogott: [C: 03+2] dns-floating-ip-updater: handle addresses from Neutron [puppet] - 10https://gerrit.wikimedia.org/r/480158 (https://phabricator.wikimedia.org/T212143) (owner: 10Andrew Bogott) [19:41:31] (03PS2) 10Andrew Bogott: dns-floating-ip-updater: handle addresses from Neutron [puppet] - 10https://gerrit.wikimedia.org/r/480158 (https://phabricator.wikimedia.org/T212143) [19:47:22] !log ppchelko@deploy1001 Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts (duration: 10m 45s) [19:47:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:50:24] (03PS1) 10Ottomata: Fixing typo in presto hive.metastore.uri in cloud-analytics [puppet] - 10https://gerrit.wikimedia.org/r/480162 [19:51:13] (03CR) 10Ottomata: [C: 03+2] Fixing typo in presto hive.metastore.uri in cloud-analytics [puppet] - 10https://gerrit.wikimedia.org/r/480162 (owner: 10Ottomata) [20:00:31] (03CR) 10BBlack: New zone generator gen-zones.py (032 comments) [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [20:01:20] (03PS4) 10BBlack: New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 [20:01:36] (03CR) 10jerkins-bot: [V: 04-1] New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [20:02:45] !log remove sandbox-out4 from ulsfo - T212155 [20:02:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:05:09] @seen legoktm [20:05:09] mutante: legoktm is in here, right now [20:05:11] !log remove sandbox-out4 from all routers - T212155 [20:05:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:11:12] (03PS1) 10Bstorm: sonofgridengine: make shadow more consistent with master [puppet] - 10https://gerrit.wikimedia.org/r/480164 (https://phabricator.wikimedia.org/T211258) [20:12:35] (03CR) 10Bstorm: [C: 03+2] sonofgridengine: make shadow more consistent with master [puppet] - 10https://gerrit.wikimedia.org/r/480164 (https://phabricator.wikimedia.org/T211258) (owner: 10Bstorm) [20:17:53] !log creating new ganeti VM doc1001.eqiad.wmnet for doc.wikimedia.org - specs as requested by hashar on T211974 [20:17:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:17:56] T211974: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 [20:23:59] !log ppchelko@deploy1001 Started deploy [recommendation-api/deploy@c1b6b32]: Rollback to c1b6b324015bd565 until the checks are fixed [20:24:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:25:49] 10Operations, 10serviceops, 10User-ArielGlenn, 10User-jijiki: create IRC channel for the Service Operations SRE subteam - https://phabricator.wikimedia.org/T211902 (10valhallasw) [20:25:54] 10Operations, 10serviceops, 10User-ArielGlenn, 10User-jijiki: create IRC channel for the Service Operations SRE subteam - https://phabricator.wikimedia.org/T211902 (10Dzahn) [20:25:55] !log ppchelko@deploy1001 Finished deploy [recommendation-api/deploy@c1b6b32]: Rollback to c1b6b324015bd565 until the checks are fixed (duration: 01m 56s) [20:25:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:26:38] 10Operations, 10Release Pipeline: blubber template for nodejs should allow defining configuration files to copy to the container - https://phabricator.wikimedia.org/T211580 (10mobrovac) [20:27:01] 10Operations, 10ops-eqiad, 10media-storage, 10Patch-For-Review, 10User-fgiunchedi: rack/setup/install ms-be10[44-50].eqiad.wmnet - https://phabricator.wikimedia.org/T209618 (10RobH) a:05Cmjohnson→03fgiunchedi Filippo, ms-be1048 has been relocated into row d & reimaged with its row d ip address. sho... [20:27:03] 10Operations, 10serviceops, 10User-ArielGlenn, 10User-jijiki: create IRC channel for the Service Operations SRE subteam - https://phabricator.wikimedia.org/T211902 (10valhallasw) [20:27:46] 10Operations, 10Release Pipeline: blubber template for nodejs should allow defining configuration files to copy to the container - https://phabricator.wikimedia.org/T211580 (10mobrovac) As of `services-runner` `v2.6.8`, you can now set the `APP_CONFIG_PATH` variable that specifies where the configuration file... [20:33:25] 10Operations, 10ops-codfw, 10User-fgiunchedi: ms-be2047 spontaneous reboots - https://phabricator.wikimedia.org/T209921 (10Papaul) The problem happen again twice after replacing CPU1 Normal,Mon 17 Dec 2018 20:29:29,An OEM diagnostic event occurred., Normal,Mon 17 Dec 2018 20:29:29,An OEM diagnostic event... [20:34:36] (03PS17) 10DCausse: [cirrus] Start writing to psi & omega [mediawiki-config] - 10https://gerrit.wikimedia.org/r/476271 (https://phabricator.wikimedia.org/T210381) [20:34:38] (03PS17) 10DCausse: [cirrus] Start using replica group settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/476272 (https://phabricator.wikimedia.org/T210381) [20:34:40] (03PS19) 10DCausse: [cirrus] Cleanup transitional states [mediawiki-config] - 10https://gerrit.wikimedia.org/r/476273 (https://phabricator.wikimedia.org/T210381) [21:00:04] cscott, arlolra, subbu, bearND, halfak, and Amir1: #bothumor I � Unicode. All rise for Services – Parsoid / Citoid / Mobileapps / ORES / … deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T2100). [21:12:09] 10Operations, 10ops-codfw, 10decommission, 10Discovery-Search (Current work): Decommission elastic2001-2024 - https://phabricator.wikimedia.org/T211023 (10Papaul) Before ` papaul@asw-c-codfw> show interfaces descriptions | match "ge-1/0/1[0-2]" ge-1/0/10 up down elastic2013 ge-1/0/11 u... [21:14:24] 10Operations, 10ops-codfw, 10decommission, 10Discovery-Search (Current work): Decommission elastic2001-2024 - https://phabricator.wikimedia.org/T211023 (10Papaul) [21:15:46] !log mholloway-shell@deploy1001 Started deploy [mobileapps/deploy@92fdf43]: Update mobileapps to d244439 [21:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:18:26] 10Operations, 10TechCom, 10Core Platform Team (Session Management Service (CDP2)), 10Core Platform Team Backlog (Next), and 4 others: Establish an SLA for session storage - https://phabricator.wikimedia.org/T211721 (10Imarlier) [21:20:07] (03PS3) 10Cwhite: hiera: add puppetboard and puppetdb to puppet cluster [puppet] - 10https://gerrit.wikimedia.org/r/479772 (https://phabricator.wikimedia.org/T210486) [21:20:17] !log mholloway-shell@deploy1001 Finished deploy [mobileapps/deploy@92fdf43]: Update mobileapps to d244439 (duration: 04m 31s) [21:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:20:20] (03PS1) 10Ottomata: Fix Hadoop settings for presto in cloud-analytics [puppet] - 10https://gerrit.wikimedia.org/r/480243 (https://phabricator.wikimedia.org/T204951) [21:20:40] (03PS3) 10Cwhite: profile: enable statsd_exporter and add matching rules to ores::worker [puppet] - 10https://gerrit.wikimedia.org/r/479563 (https://phabricator.wikimedia.org/T205870) [21:21:22] (03PS2) 10Ottomata: Fix Hadoop settings for presto in cloud-analytics [puppet] - 10https://gerrit.wikimedia.org/r/480243 (https://phabricator.wikimedia.org/T204951) [21:21:26] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Fix Hadoop settings for presto in cloud-analytics [puppet] - 10https://gerrit.wikimedia.org/r/480243 (https://phabricator.wikimedia.org/T204951) (owner: 10Ottomata) [21:23:10] (03PS4) 10Cwhite: profile: enable statsd_exporter and add matching rules to ores::worker [puppet] - 10https://gerrit.wikimedia.org/r/479563 (https://phabricator.wikimedia.org/T205870) [21:23:18] (03CR) 10Cwhite: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/13978/" [puppet] - 10https://gerrit.wikimedia.org/r/479563 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [21:25:59] (03CR) 10Jdlrobson: [C: 04-1] "Should be abandoned. This is the default and MobileFrontend shouid never change this default." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479250 (https://phabricator.wikimedia.org/T210390) (owner: 10Pmiazga) [21:27:43] (03PS1) 10Ottomata: Remove specific settings from profile::hadoop::common and put into hiera [puppet] - 10https://gerrit.wikimedia.org/r/480245 [21:28:38] (03CR) 10jerkins-bot: [V: 04-1] Remove specific settings from profile::hadoop::common and put into hiera [puppet] - 10https://gerrit.wikimedia.org/r/480245 (owner: 10Ottomata) [21:30:31] (03PS2) 10Ottomata: Remove specific settings from profile::hadoop::common and put into hiera [puppet] - 10https://gerrit.wikimedia.org/r/480245 [21:32:34] (03CR) 10Gergő Tisza: "mediawiki-config is one of the few repos where +2 only happens as part of the deployment process (there are no release branches, it is dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479997 (https://phabricator.wikimedia.org/T212099) (owner: 10Gergő Tisza) [21:32:45] (03PS2) 10Gergő Tisza: Disable MediaViewer thumbnail URL guessing for private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479997 (https://phabricator.wikimedia.org/T212099) [21:33:10] !log arlolra@deploy1001 Started deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to 4eba44e [21:33:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:34:49] 10Operations, 10serviceops, 10User-ArielGlenn, 10User-jijiki: create IRC channel for the Service Operations SRE subteam - https://phabricator.wikimedia.org/T211902 (10Dzahn) [21:36:30] (03CR) 10Ottomata: "In https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/402425/ I added this to profile::hadoop::common which is not a good place for it" [puppet] - 10https://gerrit.wikimedia.org/r/480245 (owner: 10Ottomata) [21:36:46] (03CR) 10Ottomata: "https://puppet-compiler.wmflabs.org/compiler1002/13980/an-master1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/480245 (owner: 10Ottomata) [21:38:47] 10Operations, 10Analytics, 10Security-Team, 10WMF-Legal, 10Software-Licensing: Can exfat be used in WMF production? - https://phabricator.wikimedia.org/T210667 (10chasemp) 05Open→03Resolved a:03chasemp [21:40:28] (03PS1) 10Dzahn: install_server: add doc1001 to DHCP and netboot [puppet] - 10https://gerrit.wikimedia.org/r/480247 (https://phabricator.wikimedia.org/T211974) [21:40:49] (03PS1) 10Rush: stat: add exfat for temporary narrow and approved workflow [puppet] - 10https://gerrit.wikimedia.org/r/480248 (https://phabricator.wikimedia.org/T211327) [21:41:50] (03CR) 10Dzahn: [C: 03+2] install_server: add doc1001 to DHCP and netboot [puppet] - 10https://gerrit.wikimedia.org/r/480247 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [21:41:57] (03PS2) 10Dzahn: install_server: add doc1001 to DHCP and netboot [puppet] - 10https://gerrit.wikimedia.org/r/480247 (https://phabricator.wikimedia.org/T211974) [21:42:05] !log arlolra@deploy1001 Finished deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to 4eba44e (duration: 08m 55s) [21:42:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:43:17] (03Abandoned) 10Pmiazga: Define the default mobile content provider [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479250 (https://phabricator.wikimedia.org/T210390) (owner: 10Pmiazga) [21:47:16] (03CR) 10Ottomata: [C: 03+1] stat: add exfat for temporary narrow and approved workflow [puppet] - 10https://gerrit.wikimedia.org/r/480248 (https://phabricator.wikimedia.org/T211327) (owner: 10Rush) [21:49:34] (03PS2) 10Rush: stat: add exfat for temporary narrow and approved workflow [puppet] - 10https://gerrit.wikimedia.org/r/480248 (https://phabricator.wikimedia.org/T211327) [21:50:36] (03CR) 10Rush: [C: 03+2] stat: add exfat for temporary narrow and approved workflow [puppet] - 10https://gerrit.wikimedia.org/r/480248 (https://phabricator.wikimedia.org/T211327) (owner: 10Rush) [21:55:24] !log Updated Parsoid to 4eba44e (T204622, T211941) [21:55:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:55:29] T204622: Use native Javascript (ES6) classes instead of prototype-based definition pattern in the Parsoid codebase - https://phabricator.wikimedia.org/T204622 [21:55:29] T211941: node.getAttribute is not a function - https://phabricator.wikimedia.org/T211941 [21:58:43] !log stat1004:/var/log# mkfs.exfat /dev/sde && mkdir /mnt/T211327 && mount /dev/sde /mnt/T211327/ [21:58:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:00:04] bawolff and Reedy: My dear minions, it's time we take the moon! Just kidding. Time for Weekly Security deployment window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181217T2200). [22:00:27] !log puppetmaster - signed cert for doc1001 (ganeti VM), initial puppet run [22:00:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:06:03] (03PS1) 10Herron: logstash::collector: pin curator to components/spicerack on stretch [puppet] - 10https://gerrit.wikimedia.org/r/480252 (https://phabricator.wikimedia.org/T211859) [22:06:14] 10Operations, 10Product-Analytics, 10Patch-For-Review: Upload shiny-server .deb to our Stretch apt repository - https://phabricator.wikimedia.org/T168967 (10mpopov) 05Open→03Resolved It works! :D Thanks, @aborrero! [22:06:49] 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-fgiunchedi, 10User-herron: cronspam from elasticsearch-curator on stretch - https://phabricator.wikimedia.org/T211859 (10herron) Looks like we have updated `elasticsearch-curator` stretch packages already available in `component/spicerack` `... [22:10:40] 10Operations, 10hardware-requests: Procure logstash hardware in eqiad - https://phabricator.wikimedia.org/T210498 (10colewhite) p:05Triage→03Normal [22:11:16] 10Operations, 10Operations-Software-Development: Decommission script race condition - https://phabricator.wikimedia.org/T206448 (10colewhite) p:05Triage→03Normal [22:11:47] 10Operations, 10Wikimedia-Logstash, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, and 3 others: Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal) - https://phabricator.wikimedia.org/T205849 (10herron) [22:11:49] 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review: Procure and provision Logging pipeline hardware in multiple datacenters - https://phabricator.wikimedia.org/T205850 (10herron) 05Open→03Resolved done! [22:24:38] fatal: Stray .git/rebase-apply directory found. [22:24:51] it's empty, though [22:26:32] also, ls claims it was created by me, right now. wtf? [22:27:37] (03PS1) 10Dzahn: create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) [22:28:14] (03CR) 10jerkins-bot: [V: 04-1] create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [22:28:50] 10Operations, 10serviceops: Add `supervised` option to redis configuration - https://phabricator.wikimedia.org/T212102 (10colewhite) p:05Triage→03Normal [22:30:46] (03PS2) 10Dzahn: create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) [22:31:23] (03CR) 10jerkins-bot: [V: 04-1] create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [22:39:09] (03PS3) 10Dzahn: create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) [22:39:46] (03CR) 10jerkins-bot: [V: 04-1] create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [22:44:20] (03PS4) 10Dzahn: create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) [22:48:43] (03CR) 10Paladox: [C: 03+1] create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [22:50:19] (03CR) 10Dzahn: [C: 03+2] create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [22:50:35] (03PS5) 10Dzahn: create skeleton role for docs.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/480254 (https://phabricator.wikimedia.org/T211974) [22:59:31] (03PS1) 10Cwhite: proton: enable statsd_exporter and add matching rules to profile::proton [puppet] - 10https://gerrit.wikimedia.org/r/480259 (https://phabricator.wikimedia.org/T205870) [23:00:50] (03PS5) 10BBlack: New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 [23:00:52] (03PS1) 10BBlack: Minor improvements to check-gdnsd [dns] - 10https://gerrit.wikimedia.org/r/480260 [23:01:06] (03CR) 10jerkins-bot: [V: 04-1] New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [23:01:10] (03CR) 10jerkins-bot: [V: 04-1] Minor improvements to check-gdnsd [dns] - 10https://gerrit.wikimedia.org/r/480260 (owner: 10BBlack) [23:01:24] (03CR) 10Cwhite: "set wip - need to handle gc and heap metrics" [puppet] - 10https://gerrit.wikimedia.org/r/480259 (https://phabricator.wikimedia.org/T205870) (owner: 10Cwhite) [23:02:19] (03PS1) 10Dzahn: add IPv6 records for doc1001.eqiad.wmnet. [dns] - 10https://gerrit.wikimedia.org/r/480261 (https://phabricator.wikimedia.org/T211974) [23:02:34] (03CR) 10jerkins-bot: [V: 04-1] add IPv6 records for doc1001.eqiad.wmnet. [dns] - 10https://gerrit.wikimedia.org/r/480261 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:04:08] ERROR: CI tests failed: utils/check-gdnsd.sh | zone-validator[INFO] RESULT: 0 Errors, 2257 Warnings, [23:04:14] is that expected ? [23:04:51] ah, no :) [23:05:23] (03PS2) 10Dzahn: add IPv6 records for doc1001.eqiad.wmnet. [dns] - 10https://gerrit.wikimedia.org/r/480261 (https://phabricator.wikimedia.org/T211974) [23:07:02] (03CR) 10Dzahn: [C: 03+2] add IPv6 records for doc1001.eqiad.wmnet. [dns] - 10https://gerrit.wikimedia.org/r/480261 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:12:11] (03PS1) 10Bstorm: sonofgridengine: restrict the cluster from initializing itself [puppet] - 10https://gerrit.wikimedia.org/r/480264 (https://phabricator.wikimedia.org/T212153) [23:14:35] (03CR) 10Bstorm: [C: 03+2] sonofgridengine: restrict the cluster from initializing itself [puppet] - 10https://gerrit.wikimedia.org/r/480264 (https://phabricator.wikimedia.org/T212153) (owner: 10Bstorm) [23:17:23] !log launch gpg jobs for files on labstore1007 [23:17:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:17:28] !log launch gpg jobs for files on stat1004 [23:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:21:01] (03CR) 10BBlack: "recheck" [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [23:21:14] (03CR) 10jerkins-bot: [V: 04-1] New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [23:21:29] !log deployed security patch per T207750#4829275 [23:22:07] (03PS8) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [23:23:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:29:42] doing some live debugging on mwdebug1002 [23:30:50] well, deploy1001 abd mwdebug1002 [23:38:08] (03PS1) 10Dzahn: add httpd for doc.wm.org, using php-fpm, php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) [23:39:09] (03CR) 10jerkins-bot: [V: 04-1] add httpd for doc.wm.org, using php-fpm, php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:40:04] (03CR) 10Paladox: add httpd for doc.wm.org, using php-fpm, php7.2 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:46:09] 10Operations, 10DBA, 10Jade, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Here are some example queries to help with reviewing the DDL. @Marostegui, I'm especially interested in your feedback o... [23:46:29] (03CR) 10Paladox: add httpd for doc.wm.org, using php-fpm, php7.2 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:52:25] (03CR) 10Paladox: add httpd for doc.wm.org, using php-fpm, php7.2 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:52:38] (03PS2) 10Dzahn: add httpd for doc.wm.org, using php-fpm, php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) [23:53:49] (03CR) 10jerkins-bot: [V: 04-1] add httpd for doc.wm.org, using php-fpm, php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:54:32] (03CR) 10Dzahn: add httpd for doc.wm.org, using php-fpm, php7.2 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn) [23:56:39] (03PS6) 10Tim Starling: Refactor profiler.php and X-Wikimedia-Debug parsing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477939 [23:58:05] (03PS3) 10Dzahn: add httpd for doc.wm.org, using php-fpm, php7.2 [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) [23:58:32] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/480270 (https://phabricator.wikimedia.org/T211974) (owner: 10Dzahn)