[00:31:46] (03PS1) 10Tim Starling: Remove incorrect use of X-Forwarded-For [puppet] - 10https://gerrit.wikimedia.org/r/443871 (https://phabricator.wikimedia.org/T198570) [00:33:22] (03CR) 10Tim Starling: [C: 032] "I already tested this in production and it fixed the log." [puppet] - 10https://gerrit.wikimedia.org/r/443871 (https://phabricator.wikimedia.org/T198570) (owner: 10Tim Starling) [00:36:01] (03CR) 10Paladox: "See https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443665/" [puppet] - 10https://gerrit.wikimedia.org/r/443871 (https://phabricator.wikimedia.org/T198570) (owner: 10Tim Starling) [00:41:56] (03CR) 10Krinkle: [C: 032] Improve file-level documentation for various wmf-config files [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443870 (owner: 10Krinkle) [00:43:24] (03Merged) 10jenkins-bot: Improve file-level documentation for various wmf-config files [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443870 (owner: 10Krinkle) [00:46:11] (03PS1) 10Krinkle: services: Define dc-pairs of the same service together [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443872 [00:46:13] (03PS1) 10Krinkle: services: Convert LabsServices.php to static array file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443873 [00:46:15] (03PS1) 10Krinkle: services: Convert ProductionServices.php to static array file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443874 [00:46:38] !log krinkle@deploy1001 Synchronized wmf-config/: Ia3bef874a (duration: 00m 51s) [00:46:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:50:27] (03PS1) 10Krinkle: Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 [00:51:23] (03PS2) 10Krinkle: Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 [00:51:30] * Krinkle no longer staging/deploying [00:55:27] (03CR) 10Tim Starling: [C: 04-1] "I didn't see this change until after I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/443871 . Andre pointed me to T198570 an" [puppet] - 10https://gerrit.wikimedia.org/r/443665 (owner: 1020after4) [02:22:08] !log l10nupdate@deploy1001 scap sync-l10n completed (1.32.0-wmf.10) (duration: 08m 46s) [02:22:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:26:58] (03PS1) 10Krinkle: Remove $wgEnotifUseJobQ setting (removed in 2015 with MW 1.27) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443878 [02:28:58] PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 90794392 [02:30:08] RECOVERY - Postgres Replication Lag on maps2004 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 678360 [02:32:06] (03PS1) 10Krinkle: Remove $wgCentralGeoScriptURL setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443879 [02:32:30] !log l10nupdate@deploy1001 ResourceLoader cache refresh completed at Thu Jul 5 02:32:29 UTC 2018 (duration 10m 21s) [02:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:34:40] (03CR) 10Krinkle: [C: 032] Remove $wgCentralGeoScriptURL setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443879 (owner: 10Krinkle) [02:34:44] (03CR) 10Krinkle: [C: 032] Remove $wgEnotifUseJobQ setting (removed in 2015 with MW 1.27) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443878 (owner: 10Krinkle) [02:35:56] (03Merged) 10jenkins-bot: Remove $wgEnotifUseJobQ setting (removed in 2015 with MW 1.27) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443878 (owner: 10Krinkle) [02:36:11] (03Merged) 10jenkins-bot: Remove $wgCentralGeoScriptURL setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443879 (owner: 10Krinkle) [02:36:11] * Krinkle staging on deploy1001/mwdebug1002 [02:40:26] !log krinkle@deploy1001 Synchronized wmf-config/CommonSettings.php: 370242d13 and 45f4f61c2 (duration: 00m 52s) [02:40:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:58:46] (03PS1) 10Krinkle: Set $wgMediaViewer* vars directly (1/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443880 [02:58:48] (03PS1) 10Krinkle: Set $wgMediaViewer* vars directly (2/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443881 [02:58:50] (03PS1) 10Krinkle: Set $wgMediaViewer* vars directly (3/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443882 [03:01:12] * Krinkle unlocks deploy handle [03:03:42] (03PS1) 10Krinkle: Remove $wgVaryOnXFPForAPI assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443883 [03:14:11] (03PS1) 10Krinkle: Remove $wgRecentEchoInstall assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443884 [03:57:59] PROBLEM - Disk space on maps1001 is CRITICAL: DISK CRITICAL - free space: /srv 54577 MB (3% inode=99%) [04:02:28] RECOVERY - Disk space on maps1001 is OK: DISK OK [04:42:04] !log Deploy schema change on s7 primary master (db1062) T191316 T192926 T195193 [04:42:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:42:09] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [04:42:10] T195193: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 [04:42:10] T191316: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 [04:43:29] (03PS1) 10Marostegui: db-eqiad.php: Restore db1089 original weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443885 [04:47:28] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Restore db1089 original weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443885 (owner: 10Marostegui) [04:48:46] (03Merged) 10jenkins-bot: db-eqiad.php: Restore db1089 original weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443885 (owner: 10Marostegui) [04:50:00] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Restore db1089 original traffic (duration: 00m 51s) [04:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:57:16] !log Deploy schema change on s5 codfw host by host T146591 T197891 T196379 [04:57:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:57:22] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [04:57:22] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [04:57:22] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [05:02:44] (03CR) 10Tim Starling: [C: 031] "Looks good, do you want to self-merge it?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [05:29:07] !log Deploy schema change on s3 primary master (db1075) T191316 T192926 T195193 [05:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:29:13] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [05:29:13] T195193: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 [05:29:13] T191316: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 [05:44:45] (03PS1) 10Marostegui: db-eqiad.php: Depool db1100 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443890 (https://phabricator.wikimedia.org/T146591) [05:50:34] !log Deploy schema change on dbstore1002:s5 T146591 T197891 T196379 [05:50:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:50:40] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [05:50:40] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [05:50:40] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [05:56:20] 10Operations, 10Analytics, 10Analytics-EventLogging, 10EventBus, and 2 others: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10Joe) [06:08:30] (03CR) 10jenkins-bot: Remove wmgUseClusterFileBackend (2/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443534 (owner: 10Krinkle) [06:08:55] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1100 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443890 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [06:09:48] !log Optimize dewiki.logging on db1100 and dbstore1002:s5 - T197459 [06:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:09:52] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [06:10:15] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1100 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443890 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [06:11:24] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 52s) [06:11:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:11:41] !log Deploy schema change on db1100 T146591 T197891 T196379 [06:11:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:11:46] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:11:46] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:11:47] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:12:01] (03PS1) 10Elukey: profile::analytics::database::meta: add socket auth option [puppet] - 10https://gerrit.wikimedia.org/r/443893 [06:12:11] marostegui: o/ - if you have time... -^ [06:12:19] let me see [06:13:28] do you remember when we discussed about the anaytics1003 backup? It is done via mariadb::mylvmbackup [06:13:44] so not simply taking the lvm snapshot [06:14:15] but now I need to allow it to authenticate via socket [06:14:49] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler02/11673/analytics1003.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/443893 (owner: 10Elukey) [06:16:39] (03CR) 10Marostegui: [C: 031] "This looks good." [puppet] - 10https://gerrit.wikimedia.org/r/443893 (owner: 10Elukey) [06:16:51] elukey: ^ check all the comments I did with the +1 [06:17:05] and read the proposed procedure [06:22:30] marostegui: for the granting part, is it a simple grant ending with IDENTIFIED VIA unix_socket ? [06:22:37] (also thanks a lot for the tips) [06:22:40] yeah [06:23:06] super [06:24:54] Not sure if the socket is already installed there, but before stopping mysql you can do: INSTALL PLUGIN unix_socket SONAME 'auth_socket'; [06:26:46] yep I think it is not listed among the show plugins; [06:27:08] then yeah [06:28:19] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/default/nfs-common] [06:35:35] !log kartik@deploy1001 Started deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 (T191124, T198779) [06:35:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:35:40] T198779: Matxin MT outputs the delimiters used in the MT client - https://phabricator.wikimedia.org/T198779 [06:35:40] T191124: CX2: Content isn't properly divided into sections - https://phabricator.wikimedia.org/T191124 [06:39:00] (03Abandoned) 10Elukey: Ensure existence of environment conf file [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/426918 (https://phabricator.wikimedia.org/T182924) (owner: 10Elukey) [06:39:28] !log kartik@deploy1001 Finished deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 (T191124, T198779) (duration: 03m 52s) [06:39:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:50:12] (03CR) 10Elukey: "The code looks good but doesn't show the diff to main.conf (I also checked in the change d catalog for one host). Just triple checking tha" [puppet] - 10https://gerrit.wikimedia.org/r/443842 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [06:51:42] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1100" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443896 [06:55:12] (03PS2) 10Ema: cache_misc: decommission cp3009 [puppet] - 10https://gerrit.wikimedia.org/r/443827 (https://phabricator.wikimedia.org/T148422) [06:55:38] PROBLEM - Host maps-test2003 is DOWN: PING CRITICAL - Packet loss = 100% [06:56:25] (03CR) 10Ema: [C: 032] cache_misc: decommission cp3009 [puppet] - 10https://gerrit.wikimedia.org/r/443827 (https://phabricator.wikimedia.org/T148422) (owner: 10Ema) [06:58:58] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [07:03:43] (03CR) 10Elukey: mediawiki: add vhost define (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/439893 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:10:40] (03CR) 10Ema: [C: 032] Remove mgmt DNS entries for esams ex-cache_maps [dns] - 10https://gerrit.wikimedia.org/r/443851 (https://phabricator.wikimedia.org/T167376) (owner: 10Ema) [07:12:09] nice to see no www-data jobs running on terbium anymore except some idle hhvms [07:13:36] !log stop mariadb on analytics1003 to apply https://gerrit.wikimedia.org/r/443893 and enable auth via unix socket [07:13:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:14:48] (03CR) 10Elukey: [C: 032] profile::analytics::database::meta: add socket auth option [puppet] - 10https://gerrit.wikimedia.org/r/443893 (owner: 10Elukey) [07:14:54] (03PS2) 10Elukey: profile::analytics::database::meta: add socket auth option [puppet] - 10https://gerrit.wikimedia.org/r/443893 [07:15:10] (03PS1) 10Ema: Remove prod and mgmt entries for cp3009 [dns] - 10https://gerrit.wikimedia.org/r/443897 (https://phabricator.wikimedia.org/T148422) [07:18:38] (03CR) 10Ema: [C: 032] Remove prod and mgmt entries for cp3009 [dns] - 10https://gerrit.wikimedia.org/r/443897 (https://phabricator.wikimedia.org/T148422) (owner: 10Ema) [07:22:13] (03CR) 10Giuseppe Lavagetto: "yes, that's expected as the old thing was a file, not a template. The content of the vhosts have been copied verbatim to the new template." [puppet] - 10https://gerrit.wikimedia.org/r/443842 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:23:30] (03CR) 10Muehlenhoff: mediawiki: add vhost define (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/439893 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:25:07] <_joe_> moritzm, elukey the whole idea is to substitute progressively all vhosts for the wikis with mediawiki::vhost definitions [07:25:26] <_joe_> so that finally the whole thing is standardized [07:25:38] yeah, sounds great to me [07:25:44] <_joe_> and can be later exported as a logic to a PHP entrypoint for MediaWiki [07:25:57] <_joe_> of course we will need to do this in a painfully slow way [07:26:06] <_joe_> and test each individual vhost [07:28:01] (03CR) 10Vgutierrez: [C: 031] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/433928 (owner: 10Volans) [07:28:22] yep yep, I saw some differences in the vhost definition and wanted to ask, I think it is a awesome effort [07:29:22] <_joe_> I mean we can also go the easier rout [07:29:26] <_joe_> *route [07:29:37] <_joe_> which is, we transform all the sites defs into templates [07:30:09] <_joe_> and modify just the bits about the redirect to fcgi [07:30:33] <_joe_> I'm open to doing that instead if you think that's the better route [07:32:17] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1100" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443896 (owner: 10Marostegui) [07:33:36] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1100" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443896 (owner: 10Marostegui) [07:33:58] RECOVERY - Host maps-test2003 is UP: PING OK - Packet loss = 0%, RTA = 36.92 ms [07:34:42] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 52s) [07:34:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:35:39] (03PS1) 10Marostegui: db-eqiad.php: Depool db1096:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443899 (https://phabricator.wikimedia.org/T146591) [07:35:42] <_joe_> anyways, I'm going to try to run the first of those patches if any of you gives me a +1 [07:36:32] sounds good, I'll have a more complete look in a bit [07:36:42] (03PS1) 10Elukey: analytics-meta.my.cnf.production.erb: set the same socket/port for client [puppet] - 10https://gerrit.wikimedia.org/r/443900 [07:37:09] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1096:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443899 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:37:27] (03CR) 10Elukey: [C: 032] analytics-meta.my.cnf.production.erb: set the same socket/port for client [puppet] - 10https://gerrit.wikimedia.org/r/443900 (owner: 10Elukey) [07:38:27] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1096:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443899 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:39:31] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 50s) [07:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:39:34] !log Deploy schema change on db1096:3315 T146591 T197891 T196379 [07:39:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:39:39] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [07:39:39] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [07:39:40] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [07:40:12] !log Optimize dewiki.logging on db1096:3315 - T197459 [07:40:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:15] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [07:41:38] (03PS1) 10Elukey: Revert "Explicitly set database password for hive/oozie" [puppet] - 10https://gerrit.wikimedia.org/r/443902 [07:42:03] (03PS2) 10Elukey: Revert "Explicitly set database password for hive/oozie" [puppet] - 10https://gerrit.wikimedia.org/r/443902 [07:42:42] (03CR) 10Elukey: [C: 032] Revert "Explicitly set database password for hive/oozie" [puppet] - 10https://gerrit.wikimedia.org/r/443902 (owner: 10Elukey) [08:02:44] (03PS1) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [08:02:46] (03PS1) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [08:02:48] (03PS1) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [08:02:50] (03PS1) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [08:03:07] (03CR) 10jerkins-bot: [V: 04-1] reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:03:20] (03CR) 10jerkins-bot: [V: 04-1] reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:04:18] (03PS2) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [08:05:18] (03PS2) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [08:08:00] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me, this will allow for some considerable cleanups" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/439893 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:09:19] (03PS1) 10Elukey: profile::analytics::database::meta::backup: set socket [puppet] - 10https://gerrit.wikimedia.org/r/443909 [08:09:58] (03CR) 10Elukey: [C: 032] profile::analytics::database::meta::backup: set socket [puppet] - 10https://gerrit.wikimedia.org/r/443909 (owner: 10Elukey) [08:10:53] <_joe_> moritzm, elukey I'm gonna start deploying this then [08:10:55] (03PS2) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [08:11:03] (03PS2) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [08:12:30] ack [08:12:48] !log draining restbase2001 for reboot to pick up Intel microcode update [08:12:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:13:30] (03PS3) 10Giuseppe Lavagetto: mediawiki::web: move mediawiki.org, test.wikidata to individual files [puppet] - 10https://gerrit.wikimedia.org/r/443842 (https://phabricator.wikimedia.org/T196968) [08:13:50] this == --^ ? If so, +1 :) [08:15:25] <_joe_> elukey: yes [08:15:53] <_joe_> I created a set of test URLs for this specifically [08:15:55] (03CR) 10Elukey: [C: 031] mediawiki::web: move mediawiki.org, test.wikidata to individual files [puppet] - 10https://gerrit.wikimedia.org/r/443842 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:16:38] <_joe_> ok let's try this [08:16:56] <_joe_> but for the next patches, I'll adopt another approach [08:17:14] +2 merge and run puppet on all mw hosts at once? [08:17:16] :D [08:17:21] <_joe_> I'm gonna put mwdebug in a separated environment for puppet, and change things there first [08:17:29] <_joe_> let them roast for a few days [08:17:36] <_joe_> then move them to the main env [08:17:45] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web: move mediawiki.org, test.wikidata to individual files [puppet] - 10https://gerrit.wikimedia.org/r/443842 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:17:53] (03CR) 10Volans: "Just two totally optional alternative possibilities ;)" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:17:58] seems nice! [08:18:02] (03PS1) 10Filippo Giunchedi: graphite: start sending metrics to graphite2003 too [puppet] - 10https://gerrit.wikimedia.org/r/443912 (https://phabricator.wikimedia.org/T196483) [08:19:13] (03PS2) 10Filippo Giunchedi: graphite: start sending metrics to graphite2003 too [puppet] - 10https://gerrit.wikimedia.org/r/443912 (https://phabricator.wikimedia.org/T196483) [08:20:53] (03CR) 10Volans: "replying to myself" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:22:06] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: fix relationship name [puppet] - 10https://gerrit.wikimedia.org/r/443913 [08:22:59] PROBLEM - puppet last run on mwdebug2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:22:59] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::prod_sites: fix relationship name [puppet] - 10https://gerrit.wikimedia.org/r/443913 (owner: 10Giuseppe Lavagetto) [08:23:49] (03PS3) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [08:23:50] (03PS3) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [08:23:52] (03PS3) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [08:23:54] (03PS3) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [08:24:08] (03PS3) 10Filippo Giunchedi: graphite: start sending metrics to graphite2003 too [puppet] - 10https://gerrit.wikimedia.org/r/443912 (https://phabricator.wikimedia.org/T196483) [08:24:12] (03CR) 10jerkins-bot: [V: 04-1] reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:24:20] (03CR) 10Filippo Giunchedi: [C: 032] graphite: start sending metrics to graphite2003 too [puppet] - 10https://gerrit.wikimedia.org/r/443912 (https://phabricator.wikimedia.org/T196483) (owner: 10Filippo Giunchedi) [08:24:22] (03CR) 10jerkins-bot: [V: 04-1] reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:25:27] (03CR) 10jenkins-bot: Improve file-level documentation for various wmf-config files [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443870 (owner: 10Krinkle) [08:25:39] (03CR) 10jenkins-bot: Remove $wgEnotifUseJobQ setting (removed in 2015 with MW 1.27) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443878 (owner: 10Krinkle) [08:25:59] (03CR) 10jenkins-bot: Remove $wgCentralGeoScriptURL setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443879 (owner: 10Krinkle) [08:26:09] (03CR) 10jenkins-bot: db-eqiad.php: Restore db1089 original weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443885 (owner: 10Marostegui) [08:26:33] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1100 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443890 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:26:50] !log add graphite2003 to carbon-c-relay frontend on graphite1001 - T196483 [08:26:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:26:54] T196483: rack/setup/install graphite2003 - https://phabricator.wikimedia.org/T196483 [08:28:08] RECOVERY - puppet last run on mwdebug2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:28:19] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1100" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443896 (owner: 10Marostegui) [08:29:19] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1096:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443899 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:29:54] (03CR) 10jenkins-bot: Move filebackend.php include towards the top near other includes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443535 (owner: 10Krinkle) [08:30:14] (03CR) 10jenkins-bot: Remove duplicate phpunit entry from composer.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443108 (owner: 10Reedy) [08:30:41] (03CR) 10jenkins-bot: Move if onto newline in FeaturedFeedsWMF.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/439438 (owner: 10Reedy) [08:33:19] (03PS4) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [08:33:21] (03PS4) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [08:33:23] (03PS4) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [08:33:25] (03PS4) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [08:34:45] (03CR) 10Ema: reload-vcl: do not include layer information in additional VCL labels (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:40:10] (03PS5) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [08:40:12] (03PS5) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [08:42:32] !log draining restbase2010 for reboot to pick up Intel microcode update [08:42:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:55:58] (03PS1) 10Jcrespo: mariadb: Allow reimage to stretch of db2052, db2040 and db2045 [puppet] - 10https://gerrit.wikimedia.org/r/443919 [08:57:37] !log draining restbase1007 for reboot to pick up Intel microcode update [08:57:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:11:17] !log drain restbase200[234] and restart cassandra to pick up updated certificates [09:11:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:11:21] (03PS1) 10Arturo Borrero Gonzalez: openstack: eqiad1: nova-api runs on control boxes [puppet] - 10https://gerrit.wikimedia.org/r/443922 (https://phabricator.wikimedia.org/T196633) [09:12:40] (03CR) 10Arturo Borrero Gonzalez: [C: 032] openstack: eqiad1: nova-api runs on control boxes [puppet] - 10https://gerrit.wikimedia.org/r/443922 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [09:13:19] PROBLEM - cassandra-a SSL 10.192.16.165:7001 on restbase2002 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [09:13:42] of course, I'll downtime [09:17:49] RECOVERY - cassandra-a SSL 10.192.16.165:7001 on restbase2002 is OK: SSL OK - Certificate restbase2002-a valid until 2020-06-24 13:01:28 +0000 (expires in 720 days) [09:23:27] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1096:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443924 [09:25:17] !log rebooting ms-fe1005 as microcode update canary [09:25:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:29:57] (03PS1) 10Arturo Borrero Gonzalez: openstack: bootstrap: nova: fix endpoints [puppet] - 10https://gerrit.wikimedia.org/r/443925 (https://phabricator.wikimedia.org/T196633) [09:38:18] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1096:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443924 (owner: 10Marostegui) [09:39:35] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1096:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443924 (owner: 10Marostegui) [09:39:47] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1096:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443924 (owner: 10Marostegui) [09:40:49] !log mobrovac@deploy1001 Started deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023 [09:40:50] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 52s) [09:40:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:40:53] T187023: Create Zotero translator for sverigesradio.se - https://phabricator.wikimedia.org/T187023 [09:40:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:40:57] !log mobrovac@deploy1001 Finished deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023 (duration: 00m 08s) [09:41:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:42:50] !log rebooting ms-be1013/1016/1028/1040 as microcode update canaries [09:42:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:46:14] (03PS1) 10Giuseppe Lavagetto: environments: introduce the new mediawiki_test environment [puppet] - 10https://gerrit.wikimedia.org/r/443926 [09:46:16] (03PS1) 10Giuseppe Lavagetto: mediawiki: use alternative module for the apache sites in the test env [puppet] - 10https://gerrit.wikimedia.org/r/443927 [09:46:37] (03PS1) 10Marostegui: db-eqiad.php: Depool db1097:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443928 (https://phabricator.wikimedia.org/T146591) [09:46:57] (03CR) 10jerkins-bot: [V: 04-1] mediawiki: use alternative module for the apache sites in the test env [puppet] - 10https://gerrit.wikimedia.org/r/443927 (owner: 10Giuseppe Lavagetto) [09:47:57] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1097:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443928 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [09:49:13] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1097:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443928 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [09:49:28] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1097:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443928 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [09:49:58] PROBLEM - Check systemd state on restbase-dev1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [09:50:22] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s) [09:50:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:29] !log Deploy schema change on db1097:3315 T146591 T197891 T196379 [09:50:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:34] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [09:50:34] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [09:50:34] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [09:50:48] !log Optimize dewiki.logging on db1097:3315 - T197459 [09:50:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:51] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [09:51:08] RECOVERY - Check systemd state on restbase-dev1004 is OK: OK - running: The system is fully operational [09:51:47] (03PS5) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [09:51:49] (03PS5) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [09:51:51] (03PS6) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [09:51:53] (03PS6) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [09:51:55] (03PS1) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [09:55:11] godog, moritzm: can you ping once you are done with reboots/restarts in the rb cluster? [09:55:59] mobrovac: yup, moritzm is done and I'll ping you once done [09:56:17] ETA is 15min [09:56:50] oh ok great [09:56:51] (03CR) 10MarcoAurelio: "Rather: https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/InitialiseSettings.php$10136-10143" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440092 (https://phabricator.wikimedia.org/T197095) (owner: 10Urbanecm) [09:56:53] thnx godog [10:00:22] (03CR) 10MarcoAurelio: "This should be reverted as the blackout ends in few minutes." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443742 (https://phabricator.wikimedia.org/T198761) (owner: 10Urbanecm) [10:03:21] (03PS2) 10Ladsgroup: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443387 (https://phabricator.wikimedia.org/T194165) [10:05:23] (03PS1) 10MarcoAurelio: Revert "Change logo for eswiki temporarily" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443933 [10:05:46] jouncebot: next [10:05:46] In 2 hour(s) and 54 minute(s): European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180705T1300) [10:05:50] (03PS2) 10Giuseppe Lavagetto: environments: introduce the new mediawiki_test environment [puppet] - 10https://gerrit.wikimedia.org/r/443926 [10:06:36] (03CR) 10Giuseppe Lavagetto: [C: 032] environments: introduce the new mediawiki_test environment [puppet] - 10https://gerrit.wikimedia.org/r/443926 (owner: 10Giuseppe Lavagetto) [10:10:38] mobrovac: {{done}} [10:10:47] gracias [10:11:17] <_joe_> puppet failures on the mwdebug servers would be due to my changes [10:12:16] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: add main modulepath to the list of modulepaths [puppet] - 10https://gerrit.wikimedia.org/r/443935 [10:12:29] PROBLEM - puppet last run on mwdebug1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:12:42] (03CR) 10Giuseppe Lavagetto: [V: 032 C: 032] mediawiki_test: add main modulepath to the list of modulepaths [puppet] - 10https://gerrit.wikimedia.org/r/443935 (owner: 10Giuseppe Lavagetto) [10:14:00] <_joe_> ah, sigh, moar failures due to our non-standard puppet configuration [10:15:18] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: add the private repo as well [puppet] - 10https://gerrit.wikimedia.org/r/443936 [10:15:28] (03CR) 10Giuseppe Lavagetto: [V: 032 C: 032] mediawiki_test: add the private repo as well [puppet] - 10https://gerrit.wikimedia.org/r/443936 (owner: 10Giuseppe Lavagetto) [10:17:38] RECOVERY - puppet last run on mwdebug1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:19:18] PROBLEM - Hue Server on thorium is CRITICAL: PROCS CRITICAL: 2 processes with command name python2.7, args /usr/lib/hue/build/env/bin/hue [10:19:45] (03PS1) 10Ladsgroup: Add "L" as an alias of Lexeme namespace in wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443937 (https://phabricator.wikimedia.org/T195493) [10:21:29] checking thorium [10:21:29] RECOVERY - Hue Server on thorium is OK: PROCS OK: 1 process with command name python2.7, args /usr/lib/hue/build/env/bin/hue [10:27:01] (03Abandoned) 10Ladsgroup: Add edit and create rate limit for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/408629 (https://phabricator.wikimedia.org/T184948) (owner: 10Ladsgroup) [10:30:15] (03PS6) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [10:30:16] (03PS6) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [10:30:18] (03PS7) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [10:30:20] (03PS7) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [10:30:22] (03PS2) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [10:30:57] (03PS1) 10Ladsgroup: Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) [10:31:08] !log installing ncurses security updates on jessie [10:31:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:06] (03CR) 10jerkins-bot: [V: 04-1] Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) (owner: 10Ladsgroup) [10:38:01] (03PS2) 10Ladsgroup: Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) [10:39:11] (03CR) 10jerkins-bot: [V: 04-1] Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) (owner: 10Ladsgroup) [10:40:23] (03PS3) 10Ladsgroup: Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) [10:51:26] (03PS4) 10Sbisson: Enable ORES wp10, draftquality on draft ns (118) for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443686 (https://phabricator.wikimedia.org/T198768) [11:05:58] !log Stopping Jenkins CI for kernel upgrade on contint1001 [11:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:06:34] !log rebooting contint1001 for kernel update/enabling microcode [11:06:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:15:59] (03PS1) 10Muehlenhoff: Enable microcode updates for CI masters [puppet] - 10https://gerrit.wikimedia.org/r/443944 (https://phabricator.wikimedia.org/T127825) [11:16:51] (03PS2) 10Urbanecm: Revert "Change logo for eswiki temporarily" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443933 (https://phabricator.wikimedia.org/T198846) (owner: 10MarcoAurelio) [11:16:59] (03CR) 10Muehlenhoff: [C: 032] Enable microcode updates for CI masters [puppet] - 10https://gerrit.wikimedia.org/r/443944 (https://phabricator.wikimedia.org/T127825) (owner: 10Muehlenhoff) [11:17:02] (03CR) 10Urbanecm: [C: 031] Revert "Change logo for eswiki temporarily" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443933 (https://phabricator.wikimedia.org/T198846) (owner: 10MarcoAurelio) [11:17:31] (03CR) 10Hashar: [C: 031] ";)" [puppet] - 10https://gerrit.wikimedia.org/r/438164 (owner: 10Muehlenhoff) [11:22:15] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1097:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443945 [11:30:56] !log installing python-mimeparse updates from jessie point release [11:30:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:33:06] (03PS2) 10Arturo Borrero Gonzalez: openstack: bootstrap: nova: fix endpoints [puppet] - 10https://gerrit.wikimedia.org/r/443925 (https://phabricator.wikimedia.org/T196633) [11:34:09] (03CR) 10Arturo Borrero Gonzalez: [C: 032] openstack: bootstrap: nova: fix endpoints [puppet] - 10https://gerrit.wikimedia.org/r/443925 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [12:08:34] (03PS1) 10Arturo Borrero Gonzalez: openstack: add openstack-cvps wrapper script [puppet] - 10https://gerrit.wikimedia.org/r/443953 (https://phabricator.wikimedia.org/T196633) [12:11:01] 10Operations: Integrate jessie 8.11 point update - https://phabricator.wikimedia.org/T198058 (10MoritzMuehlenhoff) These updates are fully rolled out: ``` base-files libipc-run-perl ncurses python-mimeparse xerces-c ``` openldap doesn't apply as we use a custom version on jessie. [12:14:51] (03CR) 10Arturo Borrero Gonzalez: [C: 032] openstack: add openstack-cvps wrapper script [puppet] - 10https://gerrit.wikimedia.org/r/443953 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [13:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) European Mid-day SWAT(Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180705T1300). [13:00:04] subbu, stephanebisson, aaron, Pchelolo, Amir1, and Hauskatze: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:13] * Hauskatze reporting for duty [13:00:15] here [13:00:28] o/ [13:00:58] * subbu is here for s/Tidy/RemexHtml/g [13:01:17] o/ [13:01:48] subbu, stephanebisson, aaron, Pchelolo, Amir1, and Hauskatze: most of you look like you are deployers ;) want to deploy your own patches? [13:02:00] I can SWAT today [13:02:19] (for people that can not deploy, or would prefer me deploying) [13:02:19] (03CR) 10Hashar: [C: 031] "Eventually I remembered I wrote a rspec-puppet test suite for the CI image (integration/config.git ./dib/puppet/). I ran it with this pa" [puppet] - 10https://gerrit.wikimedia.org/r/438164 (owner: 10Muehlenhoff) [13:02:29] zeljkof: I'm not a deployer actually I think.. [13:02:45] zeljkof: not a deployer [13:03:19] o/ [13:03:21] Pchelolo: for realz?! congrats on seniority! :) [13:03:26] I can deploy my stuff [13:03:37] Amir1: go ahead while I get ready [13:03:58] zeljkof: maybe I am, never had the need to try :) [13:04:00] (03PS3) 10Ladsgroup: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443387 (https://phabricator.wikimedia.org/T194165) [13:04:34] can someone deploy my patch? [13:05:04] (03CR) 10Ladsgroup: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443387 (https://phabricator.wikimedia.org/T194165) (owner: 10Ladsgroup) [13:05:09] Pchelolo: you are ;) https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/admin/data/data.yaml;2a75f216c11fe085c51b89fc2f74349ea60343c7$72 [13:05:15] subbu: sure [13:05:26] ty [13:05:34] no problemo [13:05:49] oh, ok, would still prefer you deploying zeljkof :) [13:05:59] PROBLEM - Host cp3048 is DOWN: PING CRITICAL - Packet loss = 100% [13:06:02] Pchelolo: no problemo, I 'm here to deploy :) [13:06:03] well, if he's never done it then better if somebody else does it for him [13:06:21] Hauskatze: there was a time I did my fist deployment... many moons ago ;) [13:06:33] it's copy pasting commands from a wiki page [13:06:45] nothing fancy if nothing goes wrong [13:06:45] it's not that easy when scap goes crazy [13:06:46] (03Merged) 10jenkins-bot: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443387 (https://phabricator.wikimedia.org/T194165) (owner: 10Ladsgroup) [13:06:52] yeah [13:07:28] Pchelolo: This is practically my step-by-step guide when I do deployment: https://wikitech.wikimedia.org/wiki/SWAT_deploys/Deployers [13:07:38] same here [13:07:44] I'm good at copy/pasting [13:07:55] {{plagiarism}} :P [13:08:28] how do you pull stuff to the mwdebugs fwiw? [13:08:29] copy/pasting from stack overflow is my highly ranked skill at linkedin :) [13:08:44] (03CR) 10jenkins-bot: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443387 (https://phabricator.wikimedia.org/T194165) (owner: 10Ladsgroup) [13:08:55] scap pull, ah, very easy [13:09:25] (03PS6) 10Aaron Schulz: Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440469 (https://phabricator.wikimedia.org/T198239) [13:09:56] subbu: please stand by, you are the first, as soon as Amir1 is done [13:09:58] works fine [13:10:03] k [13:10:03] going to prod [13:11:49] ACKNOWLEDGEMENT - Host cp3048 is DOWN: PING CRITICAL - Packet loss = 100% Ema T190607 [13:12:32] !log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:443387|Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks (T194165)]] (duration: 00m 52s) [13:12:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:12:35] T194165: Start writing to change_tag_def in production - https://phabricator.wikimedia.org/T194165 [13:13:07] my part is done, just let me know if you see anything weird [13:13:08] (03PS1) 10Vgutierrez: Ensure that depool threshold is being honored on new/updated configs [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) [13:13:22] Amir1: will do, taking over swat [13:15:00] Hello, sorry I'm late [13:15:11] subbu: I'm reviewing your commit, there seems to be some discussion, you think it's good to deploy? [13:15:31] zeljkof, yes .. compare PS1 .. PS3 ... PS1 has been +1ed .. PS3 is basically PS1. [13:15:34] stephanebisson: no problem, please stand by, you are next, after subbu [13:15:55] subbu: ok, merging, will let you know when it's at mwdebug1002 for testing [13:16:02] ok [13:16:24] (03PS4) 10Zfilipin: Replace Tidy with RemexHtml everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [13:17:03] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [13:17:14] !log Deploy schema change on db1113:3315 T146591 T197891 T196379 [13:17:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:17:20] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [13:17:20] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [13:17:21] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [13:18:24] (03Merged) 10jenkins-bot: Replace Tidy with RemexHtml everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [13:18:38] (03CR) 10jenkins-bot: Replace Tidy with RemexHtml everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [13:18:43] !log Optimize dewiki.logging on db1113:3315 - T197459 [13:18:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:18:47] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [13:19:57] subbu: your patch is at mwdebug1002, please test and let me know if I can deploy it [13:20:02] ok. [13:20:33] (03CR) 10Jcrespo: [C: 032] mariadb: Allow reimage to stretch of db2052, db2040 and db2045 [puppet] - 10https://gerrit.wikimedia.org/r/443919 (owner: 10Jcrespo) [13:20:41] (03PS2) 10Jcrespo: mariadb: Allow reimage to stretch of db2052, db2040 and db2045 [puppet] - 10https://gerrit.wikimedia.org/r/443919 [13:20:43] (03PS7) 10Aaron Schulz: Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440469 (https://phabricator.wikimedia.org/T198239) [13:21:41] zeljkof, ok .. a few pages look different in expected ways. good to go after verifying logs to make sure nothing is being logged there in terms of errors. [13:23:03] subbu: should I check the logs (they look good to me) or are you checking? [13:23:34] can you check for me? and tell me where you are looking at .. so i can keep track later as well. [13:23:55] if it looks good, yes, you can push to prod. [13:24:11] subbu: I am looking at three logstash links here https://wikitech.wikimedia.org/wiki/SWAT_deploys/Deployers#Browser_tabs [13:24:53] and at mwlog1001.eqiad.wmnet https://wikitech.wikimedia.org/wiki/SWAT_deploys/Deployers#Terminal_tabs [13:25:09] deploying [13:26:13] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:442142| Replace Tidy with RemexHtml everywhere (T175706)]] (duration: 00m 50s) [13:26:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:26:16] T175706: Progressively switch Wikimedia wikis from Tidy to RemexHTML - https://phabricator.wikimedia.org/T175706 [13:26:36] subbu: it's deployed, please test and please keep an eye on the logs for at least a few minutes [13:26:46] \o/ [13:26:49] will do. [13:27:17] stephanebisson: reviewing 443686, will ping you when it's at mwdebug1002 [13:28:20] (03PS5) 10Zfilipin: Enable ORES wp10, draftquality on draft ns (118) for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443686 (https://phabricator.wikimedia.org/T198768) (owner: 10Sbisson) [13:28:37] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443686 (https://phabricator.wikimedia.org/T198768) (owner: 10Sbisson) [13:28:48] zeljkof: I don't think I can really test on mwdebug1002... [13:29:01] stephanebisson: should I then just deploy? [13:29:36] zeljkof: ORES should start scoring new pages in Draft NS. I will be watching the logs today for unexpected error and checking the database when new pages that fit the bill are created. [13:29:52] (03Merged) 10jenkins-bot: Enable ORES wp10, draftquality on draft ns (118) for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443686 (https://phabricator.wikimedia.org/T198768) (owner: 10Sbisson) [13:29:55] zeljkof: If you are comfortable with it, I would say yes [13:30:31] stephanebisson: well, if you can't tests it at mwdebug, there is nothing else for me to do than just deploy ;) [13:31:04] (03CR) 10jenkins-bot: Enable ORES wp10, draftquality on draft ns (118) for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443686 (https://phabricator.wikimedia.org/T198768) (owner: 10Sbisson) [13:32:28] !log zfilipin@deploy1001 Synchronized wmf-config: SWAT: [[gerrit:443686|Enable ORES wp10, draftquality on draft ns (118) for enwiki (T198768)]] (duration: 00m 51s) [13:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:32:32] T198768: Store wp10 and draftquality scores for Draft namespace - https://phabricator.wikimedia.org/T198768 [13:32:49] stephanebisson: it's deployed, logs look ok so far... [13:33:00] zeljkof: thanks [13:33:47] stephanebisson: please monitor the logs for a while :) [13:34:04] AaronSchulz: around for SWAT? [13:34:26] Pchelolo: reviewing your commit, will ping you when it's at mwdebug1002 ready for testing [13:34:30] kk [13:34:34] Amir1: FYI: wp10 and draftquality scoring of Drafts is now deployed. [13:34:48] stephanebisson: Thanks for the heads up! [13:34:56] zeljkof: yeah [13:35:07] (03PS8) 10Aaron Schulz: Make test wikis just write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440469 (https://phabricator.wikimedia.org/T198239) [13:35:09] (03PS1) 10Aaron Schulz: Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443970 [13:35:15] AaronSchulz: want to deploy yourself, of should I? [13:35:48] I can [13:36:33] AaronSchulz: go ahead, I'm reviewing a patch in an extension [13:36:39] (03CR) 10Gehel: "I tried deploying on deployment-elastic07 as a test. nginx fails to start with :" [puppet] - 10https://gerrit.wikimedia.org/r/440498 (owner: 10EBernhardson) [13:36:56] Pchelolo: there is no task associated with the commit? (just asking, not required) [13:37:25] zeljkof: nope, there's no task, just some small bug we've noticed that didn't deserve a task [13:37:40] Pchelolo: ok, merging, it might take a while... [13:37:44] (03CR) 10Aaron Schulz: [C: 032] Make test wikis just write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440469 (https://phabricator.wikimedia.org/T198239) (owner: 10Aaron Schulz) [13:39:13] (03Merged) 10jenkins-bot: Make test wikis just write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440469 (https://phabricator.wikimedia.org/T198239) (owner: 10Aaron Schulz) [13:39:22] Hauskatze: please stand by, you are next, after AaronSchulz [13:39:32] zeljkof: tu sam. [13:39:41] (03CR) 10jenkins-bot: Make test wikis just write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440469 (https://phabricator.wikimedia.org/T198239) (owner: 10Aaron Schulz) [13:39:59] Hauskatze: your Croatian is getting better and better! :) [13:40:14] Pchelolo: good lock to Russian team on Saturday! ;) [13:40:20] yes it is, isn't it? :P [13:40:52] (03PS3) 10Muehlenhoff: Move php5 packages to contint class [puppet] - 10https://gerrit.wikimedia.org/r/438164 [13:40:58] zeljkof: hehe, good luck to Croatian team as well :) [13:41:20] Yep [13:41:32] !log aaron@deploy1001 Synchronized wmf-config/mc.php: Make test wikis just write to both nutcracker and mcrouter (duration: 00m 50s) [13:41:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:41:40] Pchelolo: I'm on an island in the middle of nowhere, if I was back home I would surely buy some Baltika for the game :D [13:41:52] (03CR) 10Muehlenhoff: [C: 032] Move php5 packages to contint class [puppet] - 10https://gerrit.wikimedia.org/r/438164 (owner: 10Muehlenhoff) [13:42:05] (surprisingly there is a Russian store in Zagreb) [13:42:22] AaronSchulz: done? can I continue with swat? [13:42:43] one more patch after a moment [13:42:57] (03PS2) 10Aaron Schulz: Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443970 [13:42:58] AaronSchulz: ok, let me know [13:44:15] (03CR) 10Aaron Schulz: [C: 032] Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443970 (owner: 10Aaron Schulz) [13:45:03] gerrit has inline diffs for images again!!!! https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/443933 [13:45:24] (03PS2) 10Muehlenhoff: Remove obsolete compat code for PHP 5 [puppet] - 10https://gerrit.wikimedia.org/r/438167 [13:45:58] (03Merged) 10jenkins-bot: Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443970 (owner: 10Aaron Schulz) [13:46:09] hmm, new UI or old? [13:46:21] ah, the new UI, polygerrit [13:46:27] that's it [13:46:38] ah, true [13:46:47] I was wondering when they'd add that feature [13:47:24] the old UI used to have it [13:47:36] but it got removed a while back after an update [13:47:51] (03PS3) 10Jcrespo: mariadb: Allow reimage to stretch of db2052, db2040 and db2045 [puppet] - 10https://gerrit.wikimedia.org/r/443919 [13:48:48] (03CR) 10jenkins-bot: Make mediawiki.org write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443970 (owner: 10Aaron Schulz) [13:49:54] Pchelolo: your commit is merged, will deploy it last since it's a bit more complicated than the regular config deploy [13:50:07] AaronSchulz: done? can I continue? [13:50:10] zeljkof: thank you. I'm not in a hurry [13:50:22] subbu: Yay, Remex! [13:50:29] !log installing postgresql security updates on maps cluster [13:50:29] :) [13:50:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:50:32] zeljkof: sure [13:50:41] AaronSchulz: thanks! taking over swat [13:51:05] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443933 (https://phabricator.wikimedia.org/T198846) (owner: 10MarcoAurelio) [13:51:39] Hauskatze: I need to purge logos for 443933 after deployment, right? [13:51:49] zeljkof: if you please [13:52:06] it'll speed-up things and people will stop complaining [13:52:53] (03Merged) 10jenkins-bot: Revert "Change logo for eswiki temporarily" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443933 (https://phabricator.wikimedia.org/T198846) (owner: 10MarcoAurelio) [13:52:57] Hauskatze: sure [13:52:59] on mwmaint1001 or do you still use terbium? [13:53:18] Hauskatze: hm, not sure what the docs say :) will check [13:54:16] wikitech still mentions terbium everywhere [13:54:35] terbium then it is, if it's still there :) [13:54:37] afaics on phabricator there's mwmaint1001 now and terbium has a decommission notice but should still work [13:55:06] !log zfilipin@deploy1001 Synchronized static/images/project-logos: SWAT: [[gerrit:443933|Revert "Change logo for eswiki temporarily" (T198846)]] (duration: 00m 50s) [13:55:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:55:09] T198846: Switch Spanish Wikipedia logo to normal. Protests against copyright directive are over temporarily - https://phabricator.wikimedia.org/T198846 [13:55:27] Hauskatze: it's deployed, pur-ging :D [13:55:40] :D [13:55:53] !log installing glibc updates from stretch point release [13:55:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:57:57] I see the logos rightly now [13:58:02] (03CR) 10jenkins-bot: Revert "Change logo for eswiki temporarily" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443933 (https://phabricator.wikimedia.org/T198846) (owner: 10MarcoAurelio) [13:58:14] Hauskatze: purged! https://phabricator.wikimedia.org/T198846#4399766 [13:58:56] Purr-fect [13:59:01] :D [13:59:29] Pchelolo: will ping you when your commit is at mwdebug1002 for testing. or should I just deploy? [13:59:50] zeljkof: I can actually test it [14:00:13] Pchelolo: ok, will ping you then in a few minutes when it's ready [14:01:05] (03PS2) 10Marostegui: Revert "db-eqiad.php: Depool db1097:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443945 [14:03:12] zeljkof: can we get one last in, pretty please? [14:03:29] !log stop and reimage db2052 [14:03:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:03:46] mobrovac: we are already over time, is there another deployment scheduled? [14:03:54] but sure, if it's urgent [14:04:02] nothing for the next 2h [14:04:08] Pchelolo: it's at mwdebug1002 [14:04:13] gr8, thnx [14:04:18] lemme add it to the list on wikitech [14:04:24] please do [14:04:30] mobrovac: do you want to deploy, or should I? [14:04:50] i can do it too, as you prefer/wish [14:04:51] zeljkof: checking [14:05:20] mobrovac: I prefer if devs deploy themselves ;) [14:05:29] mobrovac: I'll ping you when I'm done [14:05:36] kk zeljkof, i'll wait for your signal [14:05:40] aye aye captain! [14:05:42] thnx [14:05:57] mobrovac: watching the game on Saturday with Pchelolo? ;) [14:05:58] (03PS2) 10Aaron Schulz: Make all non-test wikis write to both nutcracker and mcrouter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440470 (https://phabricator.wikimedia.org/T198239) [14:06:21] heh zeljkof, i'm back in croatia now :( but that would be super fun [14:06:40] :D [14:07:06] zeljkof: I think it works, let's proceed [14:07:14] Pchelolo: ok, deploying [14:07:55] Can I deploy https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443945/ once that last patch is done? [14:09:27] !log zfilipin@deploy1001 Synchronized php-1.32.0-wmf.10/extensions/EventBus/: SWAT: [[gerrit:443932| Dont specify the comment if it is an empty string]] (duration: 00m 51s) [14:09:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:39] * mobrovac blocks marostegui from passing in front of the queue :P [14:09:51] mobrovac: ah sorry! missed your message :) [14:09:53] Pchelolo: it's deployed! [14:09:59] awesome. Thank yo [14:10:16] mobrovac, marostegui: I'm done, please take over :) [14:10:25] mobrovac: adelante! [14:10:40] ahorita :P [14:10:44] xdddd [14:11:35] (03PS7) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [14:11:37] (03PS7) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [14:11:39] (03PS8) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [14:11:41] (03PS8) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [14:11:43] (03PS3) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [14:11:45] (03PS1) 10Ema: cache_text: add misc-specific VTC tests [puppet] - 10https://gerrit.wikimedia.org/r/443974 (https://phabricator.wikimedia.org/T164609) [14:11:58] marostegui: uh i'm going to need a full scap sync, so go ahead in front of me [14:12:15] ok! should take a minute only for me [14:12:21] cool, ping when done [14:12:23] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1097:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443945 (owner: 10Marostegui) [14:14:16] hello jenkins, wake up please [14:14:29] xdddd [14:14:49] (03PS1) 10Jcrespo: mariadb: Allow reimage of db204X and db205X hosts only [puppet] - 10https://gerrit.wikimedia.org/r/443975 [14:15:00] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1097:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443945 (owner: 10Marostegui) [14:15:05] !log installing dbus updates from stretch point release [14:15:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:15:56] (03CR) 10Jcrespo: [C: 032] mariadb: Allow reimage of db204X and db205X hosts only [puppet] - 10https://gerrit.wikimedia.org/r/443975 (owner: 10Jcrespo) [14:16:02] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 50s) [14:16:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:16:07] mobrovac: done, all yours! [14:16:35] kk thnx marostegui [14:16:51] still waiting on jenkins ... [14:17:58] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1097:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443945 (owner: 10Marostegui) [14:27:19] PROBLEM - kubelet operational latencies on kubestage1001 is CRITICAL: instance=kubestage1001.eqiad.wmnet operation_type={create_container,start_container} https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [14:28:25] RECOVERY - kubelet operational latencies on kubestage1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [14:50:05] (03PS1) 10Marostegui: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443983 (https://phabricator.wikimedia.org/T146591) [14:50:39] (03PS1) 10Gehel: wdqs: fix logrotate configuration [puppet] - 10https://gerrit.wikimedia.org/r/443984 [14:50:58] !log installing php security updates on netmon1002 [14:51:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:51:15] (03CR) 10jerkins-bot: [V: 04-1] wdqs: fix logrotate configuration [puppet] - 10https://gerrit.wikimedia.org/r/443984 (owner: 10Gehel) [14:51:43] (03PS2) 10Gehel: wdqs: fix logrotate configuration [puppet] - 10https://gerrit.wikimedia.org/r/443984 [14:51:49] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443983 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [14:53:43] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443983 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [14:54:55] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 50s) [14:54:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:58:07] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443983 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [14:58:21] * mobrovac taking over deploy1001 for 10 mins [14:58:24] marostegui: ok with you ^ ? [14:58:24] (03CR) 10Gehel: "testing manually on wdqs1009, this seems to do the trick" [puppet] - 10https://gerrit.wikimedia.org/r/443984 (owner: 10Gehel) [14:59:25] (03CR) 10Gehel: [C: 032] wdqs: fix logrotate configuration [puppet] - 10https://gerrit.wikimedia.org/r/443984 (owner: 10Gehel) [15:00:04] !log Optimize dewiki.logging on db1082 with replication, this will generate lag on s5 on labsdb hosts/script load irssinotifier - T197459 [15:00:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:00:07] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [15:02:30] !log stop and reimage db2040 [15:02:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:03:37] !log mobrovac@deploy1001 Started scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186 [15:03:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:03:40] T198186: VisualEditor should explicitly set Accept-Language - https://phabricator.wikimedia.org/T198186 [15:13:56] (03PS1) 10Andrew Bogott: labvirt pool: replace 1001 in the pool [puppet] - 10https://gerrit.wikimedia.org/r/443994 [15:14:15] (03PS2) 10Vgutierrez: Ensure that depool threshold is being honored on new/updated configs [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) [15:16:44] (03PS1) 10Volans: Fix typo for backup1001 entries [dns] - 10https://gerrit.wikimedia.org/r/443995 (https://phabricator.wikimedia.org/T196478) [15:20:51] (03PS2) 10Andrew Bogott: labvirt pool: replace 1001 in the pool [puppet] - 10https://gerrit.wikimedia.org/r/443994 [15:22:53] <_joe_> win 32 [15:23:06] (03CR) 10Muehlenhoff: [C: 031] c-foreach-restart: Increase retry and delay defaults [debs/cassandra-tools-wmf] - 10https://gerrit.wikimedia.org/r/443848 (https://phabricator.wikimedia.org/T198787) (owner: 10Mobrovac) [15:26:58] !log upgrade (without jvm restart) prometheus-jmx-exporter on the analytics node listed in debmonitor still not running the last version [15:27:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:34:28] !log mobrovac@deploy1001 Finished scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186 (duration: 30m 50s) [15:34:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:34:31] T198186: VisualEditor should explicitly set Accept-Language - https://phabricator.wikimedia.org/T198186 [15:46:15] (03PS1) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) [15:47:58] (03CR) 10Volans: "Nice catch, adding a couple of totally optional comments" (032 comments) [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) (owner: 10Vgutierrez) [15:50:41] RECOVERY - Memory correctable errors -EDAC- on cp1053 is OK: (C)4 ge (W)2 ge 0 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=cp1053&var-datasource=eqiad%2520prometheus%252Fops [15:51:36] (03CR) 10Alexandros Kosiaris: "I am fine with the host IP change, but mgmt definitely requires chris" [dns] - 10https://gerrit.wikimedia.org/r/443995 (https://phabricator.wikimedia.org/T196478) (owner: 10Volans) [15:52:13] akosiaris: it's not a change, it was a typo in the previous CR [15:52:32] volans: so the mgmt is correct already ? [15:52:36] how do you know ? [15:52:40] check https://gerrit.wikimedia.org/r/#/c/operations/dns/+/437786/ [15:52:47] compare A vs PTR [15:53:26] ah the last octet was swapped [15:53:34] yep [15:53:53] and the swapped one conflicts with wtp1043 [15:53:54] btw [15:54:00] so it's also dangerous ;) [15:54:21] and the mgmt with dbproxy1006 [15:54:44] yep [15:54:59] (03CR) 10Alexandros Kosiaris: [C: 032] Fix typo for backup1001 entries [dns] - 10https://gerrit.wikimedia.org/r/443995 (https://phabricator.wikimedia.org/T196478) (owner: 10Volans) [15:55:24] ok merged and deployed [15:55:30] thanks a lot! [16:00:04] godog, moritzm, and _joe_: My dear minions, it's time we take the moon! Just kidding. Time for Puppet SWAT(Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180705T1600). [16:00:04] No GERRIT patches in the queue for this window AFAICS. [16:06:49] (03PS3) 10Vgutierrez: Ensure that depool threshold is being honored on new/updated configs [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) [16:07:45] (03CR) 10Vgutierrez: "@volans: thanks a lot for the review :)" (032 comments) [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) (owner: 10Vgutierrez) [16:10:28] yw :) [16:11:22] (03PS4) 10Vgutierrez: Ensure that depool threshold is being honored on new/updated configs [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) [16:13:27] !log stop and reimage db2045 [16:13:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:35] this will create lag on s8 codfw [16:14:04] (03PS15) 10Gehel: Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 (owner: 10EBernhardson) [16:14:06] (03PS18) 10Gehel: convert role::logstash::elasticsearch to profiles [puppet] - 10https://gerrit.wikimedia.org/r/441894 (owner: 10EBernhardson) [16:14:08] (03PS22) 10Gehel: prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 (owner: 10EBernhardson) [16:14:10] (03PS25) 10Gehel: Split instance define out of elasticsearch class [puppet] - 10https://gerrit.wikimedia.org/r/441338 (owner: 10EBernhardson) [16:14:12] (03PS53) 10Gehel: Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 (owner: 10EBernhardson) [16:15:16] (03CR) 10jerkins-bot: [V: 04-1] prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 (owner: 10EBernhardson) [16:22:40] (03CR) 10Gehel: [C: 04-1] Prep work for multi-instance elasticsearch refactor (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/440498 (owner: 10EBernhardson) [16:31:57] (03PS1) 10Catrope: Enable ORES edit quality filters on bswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444015 (https://phabricator.wikimedia.org/T197010) [16:38:05] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10Anomie) Added a mention in https://www.mediawiki.org/w/index.php?title=Extension:TemplateStyles&diff=2822114&oldid=2810115. [16:41:45] (03CR) 10Mark Bergsma: [C: 04-1] "This changes the semantics of what "enabled" means in pybal, which may have unintended side effects. See https://phabricator.wikimedia.org" [debs/pybal] - 10https://gerrit.wikimedia.org/r/443967 (https://phabricator.wikimedia.org/T184715) (owner: 10Vgutierrez) [16:49:47] (03PS1) 10Catrope: Enable ORES edit quality filters on srwiki (damaging only) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444018 (https://phabricator.wikimedia.org/T197012) [16:53:55] (03PS1) 10Jcrespo: mariadb: Disallow reimage of most db hosts [puppet] - 10https://gerrit.wikimedia.org/r/444021 [17:00:04] cscott, arlolra, subbu, halfak, and Amir1: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Services – Graphoid / Parsoid / Citoid / ORES. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180705T1700). [17:01:32] (03CR) 10Jcrespo: [C: 032] mariadb: Disallow reimage of most db hosts [puppet] - 10https://gerrit.wikimedia.org/r/444021 (owner: 10Jcrespo) [17:10:29] (03CR) 10Smalyshev: [C: 031] Add "L" as an alias of Lexeme namespace in wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443937 (https://phabricator.wikimedia.org/T195493) (owner: 10Ladsgroup) [17:16:22] PROBLEM - High lag on wdqs1003 is CRITICAL: 3644 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [18:00:02] PROBLEM - Check systemd state on notebook1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [18:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Your horoscope predicts another unfortunate Morning SWAT (Max 6 patches) deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180705T1800). [18:00:04] Amir1: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:03:51] 10Operations, 10Documentation: Update wikitech docs to use new mwmaint1001 instead of terbium - https://phabricator.wikimedia.org/T198805 (10Jdforrester-WMF) 05Open>03Resolved a:03Jdforrester-WMF I've done a couple of dozen edits to wikitextwiki; looks done to me. [18:03:54] 10Operations, 10Patch-For-Review: setup replacement for terbium (maintenance_server) on stretch - https://phabricator.wikimedia.org/T192092 (10Jdforrester-WMF) [18:05:50] o/ [18:05:57] sorry for being late [18:07:17] I can SWAT [18:07:58] Amir1: unless you wanted to SWAT your own stuff [18:08:03] ? [18:08:11] thcipriani: please proceed :) [18:08:17] * thcipriani does [18:09:06] (03PS2) 10Thcipriani: Add "L" as an alias of Lexeme namespace in wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443937 (https://phabricator.wikimedia.org/T195493) (owner: 10Ladsgroup) [18:09:14] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443937 (https://phabricator.wikimedia.org/T195493) (owner: 10Ladsgroup) [18:09:42] !log gehel@deploy1001 Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only) [18:09:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:10:10] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only) (duration: 00m 28s) [18:10:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:10:33] SMalyshev: ^ [18:11:03] (03Merged) 10jenkins-bot: Add "L" as an alias of Lexeme namespace in wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443937 (https://phabricator.wikimedia.org/T195493) (owner: 10Ladsgroup) [18:11:16] (03CR) 10jenkins-bot: Add "L" as an alias of Lexeme namespace in wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443937 (https://phabricator.wikimedia.org/T195493) (owner: 10Ladsgroup) [18:11:50] Amir1: ^ is live on mwdebug1002, check please [18:12:17] thcipriani: works fine! [18:12:30] cool, thanks for checking, going live [18:13:41] PROBLEM - Check systemd state on notebook1003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [18:14:44] !log thcipriani@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:443937|Add "L" as an alias of Lexeme namespace in wikidata]] T195493 (duration: 00m 59s) [18:14:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:47] (03PS4) 10Thcipriani: Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) (owner: 10Ladsgroup) [18:14:48] T195493: Set up “L” as an alias for the “Lexeme” namespace - https://phabricator.wikimedia.org/T195493 [18:14:52] ^ Amir1 should be live now [18:15:12] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) (owner: 10Ladsgroup) [18:15:18] it's great [18:16:57] (03Merged) 10jenkins-bot: Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) (owner: 10Ladsgroup) [18:17:42] Amir1: ^ is live on mwdebug1002, anything you'd like to/can test for that? [18:18:57] Testing [18:19:17] thcipriani: it's great [18:19:26] nice, ok, going live [18:19:33] it might have some performance implications but we'll see [18:21:09] (03CR) 10jenkins-bot: Set dispatchLagToMaxLagFactor to 60 for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443939 (https://phabricator.wikimedia.org/T194950) (owner: 10Ladsgroup) [18:21:19] !log thcipriani@deploy1001 Synchronized wmf-config/Wikibase-production.php: SWAT: [[gerrit:443939|Set dispatchLagToMaxLagFactor to 60 for wikidata]] T194950 (duration: 00m 51s) [18:21:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:21:22] T194950: Include Wikibase dispatch lag in API "maxlag" enforcing - https://phabricator.wikimedia.org/T194950 [18:21:25] ^ Amir1 should be live everywhere [18:21:54] works fine [18:22:38] happy to hear it :) [18:29:47] (03CR) 10Bstorm: "Shouldn't the reimage take place on one patch and the role change on the next? Or are manual changes needed for the role change to take e" [puppet] - 10https://gerrit.wikimedia.org/r/443799 (https://phabricator.wikimedia.org/T197246) (owner: 10Alexandros Kosiaris) [18:37:11] PROBLEM - High lag on wdqs1003 is CRITICAL: 3652 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [18:37:46] ^ looking into that high lag already [18:45:12] 10Operations, 10SRE-Access-Requests: Ops access request - https://phabricator.wikimedia.org/T198900 (10Cirdan) [18:48:41] !log restarting blazegraph on wdqs1003 (updater lag) [18:48:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:50:31] PROBLEM - High lag on wdqs1003 is CRITICAL: 3883 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [18:54:05] (03PS1) 10Catrope: Disable static maps on bgwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444039 [18:55:28] !log T194678 pause cirrussearch writes to codfw to check how kafka+mirrormaker responds [18:55:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:55:32] T194678: Update OtherIndex to operate on a cluster other than the one holding the wiki - https://phabricator.wikimedia.org/T194678 [18:57:43] (03CR) 10Jforrester: "> Set Ready For Review" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432702 (https://phabricator.wikimedia.org/T161553) (owner: 10Andrew Bogott) [18:58:25] !log depooling wdqs1003 to help it recover update lag [18:58:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:02:06] (03CR) 10Andrew Bogott: "not yet." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432702 (https://phabricator.wikimedia.org/T161553) (owner: 10Andrew Bogott) [19:07:19] !log T194678 un-pause cirrussearch writes to codfw [19:07:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:07:22] T194678: Update OtherIndex to operate on a cluster other than the one holding the wiki - https://phabricator.wikimedia.org/T194678 [19:12:59] (03CR) 10Eevans: [C: 032] c-foreach-restart: Increase retry and delay defaults [debs/cassandra-tools-wmf] - 10https://gerrit.wikimedia.org/r/443848 (https://phabricator.wikimedia.org/T198787) (owner: 10Mobrovac) [19:17:27] (03PS3) 10Andrew Bogott: labvirt pool: replace 1001 in the pool [puppet] - 10https://gerrit.wikimedia.org/r/443994 [19:17:56] (03PS1) 10Ottomata: Set camus kafka.move.to.earliest.offset=true [puppet] - 10https://gerrit.wikimedia.org/r/444043 (https://phabricator.wikimedia.org/T198906) [19:18:50] (03CR) 10Andrew Bogott: [C: 032] labvirt pool: replace 1001 in the pool [puppet] - 10https://gerrit.wikimedia.org/r/443994 (owner: 10Andrew Bogott) [19:19:31] (03PS2) 10Ottomata: Set camus kafka.move.to.earliest.offset=true [puppet] - 10https://gerrit.wikimedia.org/r/444043 (https://phabricator.wikimedia.org/T198906) [19:22:55] (03CR) 10Joal: "I think we should double-check that every imported data-stream has de-duplication in its first steps before applying this setting. I know " [puppet] - 10https://gerrit.wikimedia.org/r/444043 (https://phabricator.wikimedia.org/T198906) (owner: 10Ottomata) [19:23:37] (03CR) 10Ottomata: [C: 032] Set camus kafka.move.to.earliest.offset=true [puppet] - 10https://gerrit.wikimedia.org/r/444043 (https://phabricator.wikimedia.org/T198906) (owner: 10Ottomata) [19:28:11] RECOVERY - High lag on wdqs1003 is OK: (C)3600 ge (W)1200 ge 1081 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [19:30:04] !log repool wdqs1003, lag almost completely recovered [19:30:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:24:21] (03PS1) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [20:24:29] (03PS1) 10Ottomata: Camus kafka.max.historical.days=-1 for event(logging) data [puppet] - 10https://gerrit.wikimedia.org/r/444050 (https://phabricator.wikimedia.org/T198906) [20:25:13] (03PS2) 10Ottomata: Camus kafka.max.historical.days=-1 for event(logging) data [puppet] - 10https://gerrit.wikimedia.org/r/444050 (https://phabricator.wikimedia.org/T198906) [20:26:11] (03CR) 10Ottomata: [C: 032] Camus kafka.max.historical.days=-1 for event(logging) data [puppet] - 10https://gerrit.wikimedia.org/r/444050 (https://phabricator.wikimedia.org/T198906) (owner: 10Ottomata) [20:38:09] (03PS2) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [20:42:10] (03PS3) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [20:46:27] (03PS4) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [20:51:22] (03PS5) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [20:52:11] (03CR) 10jerkins-bot: [V: 04-1] Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [21:02:54] (03PS6) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:07:14] (03PS7) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:07:56] (03CR) 10jerkins-bot: [V: 04-1] Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [21:08:57] (03PS1) 10Alex Monk: phabricator: Attempt to mulitply rate limits for WMDE and WMF offices [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) [21:09:44] (03PS8) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:10:27] (03CR) 10jerkins-bot: [V: 04-1] Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [21:14:34] (03PS9) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:15:56] !log disabled unused RAID controller on labvirt1019 [21:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:18:13] (03CR) 10Alexandros Kosiaris: "Normally yes, but this box is not in service, so no harm done in bundling them." [puppet] - 10https://gerrit.wikimedia.org/r/443799 (https://phabricator.wikimedia.org/T197246) (owner: 10Alexandros Kosiaris) [21:18:15] 10Operations, 10ops-eqiad: Degraded RAID on labvirt1019 - https://phabricator.wikimedia.org/T196507 (10Bstorm) I went ahead and disabled the unused RAID controller in the BIOS. I have confirmed is not enough to clear the monitor. The lack of battery still reads as "critical". [21:27:09] Hey there, anyone here to assess possible account hijacking here? [21:27:15] (in private, if possible) [21:29:30] Urbanecm, I can't but are is this account privileged? [21:29:52] like editinterface/CU/OS or anything that kind of level [21:30:03] The account was privileged (sysop level) and resigned. [21:30:12] !log disabled unused P440ar RAID controller on labvirt1020 [21:30:12] (03PS2) 10Reedy: phabricator: Attempt to multiply rate limits for WMDE and WMF offices [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) (owner: 10Alex Monk) [21:30:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:30:16] currently privileged? [21:30:46] 10Operations, 10ops-eqiad: Degraded RAID on labvirt1019 - https://phabricator.wikimedia.org/T198918 (10ops-monitoring-bot) [21:31:33] No, only former privileges. But still trusted by others, and there are very strange things happening (I cannot talk about them publicly), that's the reason why I would not like discuss it on-chan [21:31:35] (03CR) 1020after4: "@tim: regardless of x-forwarded-for headers, this patch has other important changes." [puppet] - 10https://gerrit.wikimedia.org/r/443665 (owner: 1020after4) [21:32:20] (03PS10) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:35:37] (03PS11) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:37:07] Urbanecm, have you tried writing to trust and safety? [21:37:54] No, I haven't tried anything except writing here and to -tech [21:38:08] (and telling the story to Reedy) [21:38:24] oh, I'm sure Reedy will know what to do [21:38:36] More faith than I have [21:38:39] lol [21:40:25] lol² [21:41:01] (03PS12) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [21:43:14] (03CR) 1020after4: [C: 031] phabricator: Attempt to multiply rate limits for WMDE and WMF offices [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) (owner: 10Alex Monk) [21:43:52] PROBLEM - IPv6 ping to eqsin on ripe-atlas-eqsin IPv6 is CRITICAL: CRITICAL - failed 20 probes of 303 (alerts on 19) - https://atlas.ripe.net/measurements/11645088/#!map [21:49:01] RECOVERY - IPv6 ping to eqsin on ripe-atlas-eqsin IPv6 is OK: OK - failed 10 probes of 303 (alerts on 19) - https://atlas.ripe.net/measurements/11645088/#!map [21:50:14] (03PS6) 1020after4: phabricator: refactor preamble.php to separate unrelated functionality. [puppet] - 10https://gerrit.wikimedia.org/r/443665 [21:55:04] !log whitelist office IPs in phabricator throttle [21:55:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:55:07] (03CR) 10Andrew Bogott: "I hate how giant this is, but the puppet compiler output seems reasonable" [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [21:55:52] (03CR) 10Andrew Bogott: "I spoke too soon; labtestcontrol2001 has a diff too. I still think this is right though." [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [21:56:12] twentyafterfour: 500 errors for me from Phab following that !log [21:56:22] better now [21:56:40] bd808: yeah fixed [21:58:36] 10Operations: Setup wikimediafoundation.org domain for July 30 launch of new site - https://phabricator.wikimedia.org/T198922 (10Varnent) [21:59:38] 10Operations: Setup wikimediafoundation.org domain for July 30 launch of new site - https://phabricator.wikimedia.org/T198922 (10Varnent) Initial setup questions from Automattic: * "Would you like us to provision a free Let's Encrypt certificate for your sitewide SSL, or would you prefer to use your own certifi... [22:00:23] 10Operations: Setup wikimediafoundation.org domain for July 30 launch of new site - https://phabricator.wikimedia.org/T198922 (10Varnent) [22:43:34] (03PS7) 1020after4: phabricator: refactor preamble.php to separate unrelated functionality. [puppet] - 10https://gerrit.wikimedia.org/r/443665 [22:56:53] RoanKattouw: got an edit conflict with you on the deploy schedule [22:56:58] first time using the new conflict resolver. [22:57:01] This is amazing. [22:57:02] I love it. [22:58:03] Oh, I haven't seen it yet [22:58:09] Hopefully I will now conflict with you :D [22:59:32] Oh, no I was editing next week's table [23:00:05] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor I � Unicode. All rise for Evening SWAT (Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180705T2300). [23:00:05] RoanKattouw: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [23:00:54] I'll do the SWAT since it's all my patches anyway [23:01:04] ;) [23:01:11] Oh and Krinkle's :) [23:01:12] He goes first [23:01:34] (03CR) 10Catrope: [C: 032] Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [23:04:37] Thx [23:07:46] (03PS3) 10Catrope: Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [23:07:54] (03CR) 10Catrope: Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [23:07:57] (03CR) 10Catrope: [C: 032] Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [23:09:40] (03Merged) 10jenkins-bot: Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [23:09:53] (03CR) 10jenkins-bot: Don't use include_once for assigned values ($wgInterwikiCache) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443875 (owner: 10Krinkle) [23:10:54] Krinkle: It's on mwdebug1002 in case you want to test it there, but there might not be much to test? [23:11:28] RoanKattouw: ack, testing anyway. [23:12:36] RoanKattouw: LGTM, interwiki and inter lang links working as expected on views and edits. [23:13:09] OK, deploying [23:13:24] (03PS2) 10Catrope: Disable static maps on bgwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444039 [23:13:30] (03CR) 10Catrope: [C: 032] Disable static maps on bgwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444039 (owner: 10Catrope) [23:13:57] !log catrope@deploy1001 Synchronized wmf-config/CommonSettings.php: Use require rather than include_once for $wgInterwikiCache (duration: 00m 52s) [23:14:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:14:44] (03Merged) 10jenkins-bot: Disable static maps on bgwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444039 (owner: 10Catrope) [23:14:57] (03CR) 10jenkins-bot: Disable static maps on bgwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444039 (owner: 10Catrope) [23:19:06] !log catrope@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Disable static maps on bgwiki (duration: 00m 51s) [23:19:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:28:08] RoanKattouw: Is there more or someone else waiting to deploy? If not, was thinking of doing https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443884/ and https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443883/ [23:28:40] Go for it. I have a wmf.10 patch, but it failed Jenkins for a bogus reason and I couldn't re-+2 it until it finished failing just now [23:28:51] ( Krinkle --^^ ) [23:28:52] (03PS2) 10Krinkle: Remove $wgVaryOnXFPForAPI assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443883 [23:28:56] (03CR) 10Krinkle: [C: 032] Remove $wgVaryOnXFPForAPI assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443883 (owner: 10Krinkle) [23:29:03] (03PS2) 10Krinkle: Remove $wgRecentEchoInstall assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443884 [23:29:21] ok [23:29:30] * Krinkle staging on deploy1001/mwdebug1002 [23:30:45] (03Merged) 10jenkins-bot: Remove $wgVaryOnXFPForAPI assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443883 (owner: 10Krinkle) [23:31:23] (03CR) 10jenkins-bot: Remove $wgVaryOnXFPForAPI assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443883 (owner: 10Krinkle) [23:31:36] (03CR) 10Krinkle: [C: 032] Remove $wgRecentEchoInstall assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443884 (owner: 10Krinkle) [23:32:48] !log krinkle@deploy1001 Synchronized wmf-config/CommonSettings.php: I8165473c49 - Remove wgVaryOnXFPForAPI (duration: 00m 51s) [23:32:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:35:05] (03CR) 10Ori.livneh: [C: 031] webperf: Rename role::xenon to profile::webperf::xenon (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/443757 (https://phabricator.wikimedia.org/T195312) (owner: 10Krinkle) [23:39:13] (03CR) 10Ori.livneh: [C: 031] webperf: Rename webperf profiles for clarity (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/443752 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [23:39:47] !log krinkle@deploy1001 Synchronized wmf-config/CommonSettings.php: Ic87af972e1 - Remove wgRecentEchoInstall (duration: 00m 51s) [23:39:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:40:37] * Krinkle unlocks deploy [23:41:26] (03CR) 10jenkins-bot: Remove $wgRecentEchoInstall assignment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443884 (owner: 10Krinkle) [23:42:58] (03CR) 10Ori.livneh: [C: 031] webperf: Enable xenondata_host on perfsite in Beta Cluster [puppet] - 10https://gerrit.wikimedia.org/r/443764 (https://phabricator.wikimedia.org/T195312) (owner: 10Krinkle) [23:43:07] OK I'll deploy mine now [23:43:40] (03CR) 10Ori.livneh: [C: 031] webperf: Move site vars to profile class params (set from Hiera) [puppet] - 10https://gerrit.wikimedia.org/r/443739 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [23:47:51] !log catrope@deploy1001 Synchronized php-1.32.0-wmf.10/extensions/Echo/: Handle missing presentation model (T195253) (duration: 00m 52s) [23:47:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:47:55] T195253: Special:Notifications gives a consistent PHP exception on load ("The trash icon is not registered") for users with OpenStackManager notifications - https://phabricator.wikimedia.org/T195253 [23:56:18] (03Abandoned) 10Ori.livneh: Add a paging alert for Redis memory utilization [puppet] - 10https://gerrit.wikimedia.org/r/252396 (https://phabricator.wikimedia.org/T118331) (owner: 10Ori.livneh)