[00:12:41] subbu: sorry, i am about to depart on an airplane, but .. it's not on host bromine anymore. there are now releases1001 and releases2001 and this will be the first time they'll be used for that [00:13:15] maybe i'll have wifi [00:14:31] subbu: we need to try and replace bromine.eqiad.wmnet with releases1001.eqiad.wmnet and then see if that works, hopefully it does [00:14:43] i'll check back later / from different timezone [00:23:39] (03CR) 10Krinkle: "Verified on beta via https://en.wikipedia.beta.wmflabs.org/--errorpage-noise" [puppet] - 10https://gerrit.wikimedia.org/r/381274 (owner: 10Krinkle) [00:27:11] (03CR) 10Krinkle: [C: 04-1] "I agree this message should be localised, but I don't think this link needs to be project-specific. The point is to link to information ab" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/346264 (https://phabricator.wikimedia.org/T113114) (owner: 10Nemo bis) [00:31:04] 10Operations, 10ops-eqiad, 10Cloud-Services: rack/setup/install labvirt10(19|20).eqiad.wmnet - https://phabricator.wikimedia.org/T172538#3708653 (10RobH) a:05RobH>03chasemp [00:31:44] 10Operations, 10Cloud-Services: rack/setup/install labvirt10(19|20).eqiad.wmnet - https://phabricator.wikimedia.org/T172538#3501521 (10RobH) both of these systems are now working and calling into puppet, ready for service implementation by #cloud-services [00:36:31] PROBLEM - Check systemd state on restbase2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [00:37:00] PROBLEM - cassandra-c SSL 10.192.16.167:7001 on restbase2002 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [00:37:10] PROBLEM - cassandra-c CQL 10.192.16.167:9042 on restbase2002 is CRITICAL: connect to address 10.192.16.167 and port 9042: Connection refused [00:37:11] PROBLEM - cassandra-c service on restbase2002 is CRITICAL: CRITICAL - Expecting active but unit cassandra-c is failed [00:41:01] PROBLEM - cassandra-b CQL 10.192.16.166:9042 on restbase2002 is CRITICAL: connect to address 10.192.16.166 and port 9042: Connection refused [00:41:19] PROBLEM - cassandra-b SSL 10.192.16.166:7001 on restbase2002 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [00:48:40] RECOVERY - Check systemd state on restbase2002 is OK: OK - running: The system is fully operational [00:50:10] RECOVERY - cassandra-c SSL 10.192.16.167:7001 on restbase2002 is OK: SSL OK - Certificate restbase2002-c valid until 2018-07-19 10:52:13 +0000 (expires in 267 days) [00:50:10] RECOVERY - cassandra-b CQL 10.192.16.166:9042 on restbase2002 is OK: TCP OK - 0.036 second response time on 10.192.16.166 port 9042 [00:50:20] RECOVERY - cassandra-b SSL 10.192.16.166:7001 on restbase2002 is OK: SSL OK - Certificate restbase2002-b valid until 2018-07-19 10:52:11 +0000 (expires in 267 days) [00:57:09] RECOVERY - cassandra-c CQL 10.192.16.167:9042 on restbase2002 is OK: TCP OK - 0.037 second response time on 10.192.16.167 port 9042 [00:58:50] RECOVERY - cassandra-c service on restbase2002 is OK: OK - cassandra-c is active [01:04:10] PROBLEM - Check health of redis instance on 6380 on rdb2003 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6380 [01:04:10] PROBLEM - Check health of redis instance on 6481 on rdb2005 is CRITICAL: CRITICAL: replication_delay is 1508893446 600 - REDIS 2.8.17 on 127.0.0.1:6481 has 1 databases (db0) with 4085006 keys, up 4 minutes 3 seconds - replication_delay is 1508893446 [01:04:30] PROBLEM - Check health of redis instance on 6479 on rdb2005 is CRITICAL: CRITICAL: replication_delay is 1508893468 600 - REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 4088329 keys, up 4 minutes 26 seconds - replication_delay is 1508893468 [01:04:39] PROBLEM - Check health of redis instance on 6480 on rdb2005 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6480 [01:05:10] RECOVERY - Check health of redis instance on 6380 on rdb2003 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6380 has 1 databases (db0) with 8784001 keys, up 5 minutes 3 seconds - replication_delay is 0 [01:05:10] RECOVERY - Check health of redis instance on 6481 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6481 has 1 databases (db0) with 4079057 keys, up 5 minutes 3 seconds - replication_delay is 0 [01:05:39] RECOVERY - Check health of redis instance on 6479 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 4082218 keys, up 5 minutes 30 seconds - replication_delay is 0 [01:05:39] RECOVERY - Check health of redis instance on 6480 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6480 has 1 databases (db0) with 4080718 keys, up 5 minutes 31 seconds - replication_delay is 0 [02:18:30] (03PS1) 10Andrew Bogott: git-sync-upstream: perform rebase in a sepaarate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 [02:19:19] (03CR) 10jerkins-bot: [V: 04-1] git-sync-upstream: perform rebase in a sepaarate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (owner: 10Andrew Bogott) [02:20:04] (03PS2) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 [02:20:43] (03CR) 10jerkins-bot: [V: 04-1] git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (owner: 10Andrew Bogott) [02:23:13] (03PS3) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [02:35:30] !log l10nupdate@tin scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 59s) [02:35:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:05:27] (03CR) 10BryanDavis: git-sync-upstream: rewrite in python (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [03:15:31] !log l10nupdate@tin scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 46s) [03:15:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:16:20] PROBLEM - Router interfaces on cr1-eqord is CRITICAL: CRITICAL: host 208.80.154.198, interfaces up: 37, down: 1, dormant: 0, excluded: 0, unused: 0 [03:16:50] PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 224, down: 1, dormant: 0, excluded: 0, unused: 0 [03:22:51] !log l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 25 03:22:51 UTC 2017 (duration 7m 20s) [03:22:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:24:29] RECOVERY - Router interfaces on cr1-eqord is OK: OK: host 208.80.154.198, interfaces up: 39, down: 0, dormant: 0, excluded: 0, unused: 0 [03:24:59] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 226, down: 0, dormant: 0, excluded: 0, unused: 0 [03:26:20] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 798.65 seconds [03:42:20] (03CR) 10Andrew Bogott: git-sync-upstream: rewrite in python (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [03:51:28] (03PS3) 10Andrew Bogott: git-sync-upstream: rewrite in python [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) [03:51:30] (03PS4) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [03:57:07] (03CR) 10Jayprakash12345: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386324 (https://phabricator.wikimedia.org/T178965) (owner: 10Jon Harald Søby) [04:10:31] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 172.39 seconds [05:21:17] (03CR) 10Chad: "Do we have python-git installed on the puppetmasters already?" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [05:25:15] (03CR) 10Andrew Bogott: "> Do we have python-git installed on the puppetmasters already?" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [05:26:00] (03PS4) 10Andrew Bogott: git-sync-upstream: rewrite in python [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) [05:26:02] (03PS5) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [05:31:19] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:34:55] (03PS1) 10Marostegui: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386337 (https://phabricator.wikimedia.org/T164488) [05:37:37] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386337 (https://phabricator.wikimedia.org/T164488) (owner: 10Marostegui) [05:39:38] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386337 (https://phabricator.wikimedia.org/T164488) (owner: 10Marostegui) [05:39:47] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386337 (https://phabricator.wikimedia.org/T164488) (owner: 10Marostegui) [05:42:42] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 01m 00s) [05:42:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:42:52] T164488: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488 [05:44:40] !log Stop replication in sync on db1103 and db1077 to checksum data - T164488 [05:44:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:50:54] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1065" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386338 [05:50:59] (03PS2) 10Marostegui: Revert "db-eqiad.php: Depool db1065" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386338 [05:59:37] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1065" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386338 (owner: 10Marostegui) [06:00:06] (03PS1) 10Marostegui: Revert "db-codfw.php: Depool db2035" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386339 [06:00:10] (03PS2) 10Marostegui: Revert "db-codfw.php: Depool db2035" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386339 [06:01:18] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1065" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386338 (owner: 10Marostegui) [06:01:19] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [06:01:31] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1065" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386338 (owner: 10Marostegui) [06:01:40] (03PS3) 10Marostegui: Revert "db-codfw.php: Depool db2035" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386339 [06:02:19] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 - T174509 (duration: 00m 50s) [06:02:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:02:27] T174509: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509 [06:06:14] (03CR) 10Marostegui: [C: 032] Revert "db-codfw.php: Depool db2035" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386339 (owner: 10Marostegui) [06:07:54] (03Merged) 10jenkins-bot: Revert "db-codfw.php: Depool db2035" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386339 (owner: 10Marostegui) [06:08:03] (03CR) 10jenkins-bot: Revert "db-codfw.php: Depool db2035" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386339 (owner: 10Marostegui) [06:08:53] !log marostegui@tin Synchronized wmf-config/db-codfw.php: Repool db2035 - T178359 (duration: 00m 50s) [06:08:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:09:00] T178359: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359 [06:15:50] !log Stop MySQL on db2038 to copy its data to db2084 - T178359 [06:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:15:58] T178359: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359 [06:25:39] PROBLEM - Restbase edge esams on text-lb.esams.wikimedia.org is CRITICAL: /api/rest_v1/feed/onthisday/{type}/{mm}/{dd} (Retrieve selected the events for Jan 01) timed out before a response was received [06:26:29] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: All endpoints are healthy [06:59:50] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:05:27] !log Optimize pagelinks and templatelinks on db1021 - T174509 [07:05:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:05:35] T174509: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509 [07:09:50] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [07:40:37] !log gehel@tin Started deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842 [07:40:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:45] T167842: Find a new PIM RP IP - https://phabricator.wikimedia.org/T167842 [07:41:47] XioNoX: ^ finally getting rid of multicast for wdqs [07:42:21] !log gehel@tin Finished deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842 (duration: 01m 44s) [07:42:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:48:24] (03PS2) 10Giuseppe Lavagetto: videoscaler: decom part of the older servers [puppet] - 10https://gerrit.wikimedia.org/r/386157 [07:51:16] (03CR) 10Giuseppe Lavagetto: [C: 032] videoscaler: decom part of the older servers [puppet] - 10https://gerrit.wikimedia.org/r/386157 (owner: 10Giuseppe Lavagetto) [07:56:29] 10Operations, 10netops, 10Patch-For-Review: Find a new PIM RP IP - https://phabricator.wikimedia.org/T167842#3708969 (10Gehel) @ayounsi : wdqs should be clean of unwanted multicast. [08:02:49] <_joe_> !log decommissioning mw1168/69 as videoscalers, stopped jobrunner/jobchron, will stop hhvm/nginx/apache once current processing is done [08:02:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:17] 10Operations, 10ops-eqiad, 10User-Elukey, 10User-Joe: Decomission mw1161-69 - https://phabricator.wikimedia.org/T177387#3656944 (10Joe) [08:08:12] 10Operations, 10ops-eqiad, 10User-Elukey, 10User-Joe: Decomission mw1161-69 - https://phabricator.wikimedia.org/T177387#3656944 (10Joe) I did all the steps in decom up to the uninterruptible tasks. @Cmjohnson the servers are yours to fully decom. [08:12:30] !log upgrade mw1238-mw1258 to wikidiff2 [08:12:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:14:30] 10Operations, 10Traffic, 10Wikimedia-Logstash: Varnish does not vary elasticsearch query by request body - https://phabricator.wikimedia.org/T174960#3708988 (10ema) >>! In T174960#3705072, @EBernhardson wrote: > Actually on closer review, kibana is allowing some POST requests to a limited set of endpoints, b... [08:20:06] PROBLEM - Router interfaces on cr2-ulsfo is CRITICAL: CRITICAL: host 198.35.26.193, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0 [08:20:27] PROBLEM - Router interfaces on cr1-codfw is CRITICAL: CRITICAL: host 208.80.153.192, interfaces up: 120, down: 1, dormant: 0, excluded: 0, unused: 0 [08:26:44] (03CR) 10Volans: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/386176 (owner: 10Filippo Giunchedi) [08:29:40] (03PS2) 10Filippo Giunchedi: hieradata: gradual eqiad rollout of syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386176 [08:31:04] (03CR) 10Volans: [C: 032] PuppetDB backend: Class, Roles and Profiles shortcuts [software/cumin] - 10https://gerrit.wikimedia.org/r/384547 (https://phabricator.wikimedia.org/T178279) (owner: 10Volans) [08:31:23] (03CR) 10Filippo Giunchedi: [C: 032] hieradata: gradual eqiad rollout of syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386176 (owner: 10Filippo Giunchedi) [08:33:51] (03Merged) 10jenkins-bot: PuppetDB backend: Class, Roles and Profiles shortcuts [software/cumin] - 10https://gerrit.wikimedia.org/r/384547 (https://phabricator.wikimedia.org/T178279) (owner: 10Volans) [08:44:30] andrewbogott: I'm not sure, perhaps sum the failed events and check over a longer period of time? [08:52:15] 10Operations, 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3709050 (10MoritzMuehlenhoff) I installed the latest trusty kernels on labsdb1001/1003. [08:52:23] (03PS2) 10Gehel: role::discovery: Fix stats/ML classes [puppet] - 10https://gerrit.wikimedia.org/r/386271 (https://phabricator.wikimedia.org/T178096) (owner: 10Bearloga) [08:54:36] (03CR) 10Gehel: [C: 032] role::discovery: Fix stats/ML classes [puppet] - 10https://gerrit.wikimedia.org/r/386271 (https://phabricator.wikimedia.org/T178096) (owner: 10Bearloga) [08:58:10] (03PS2) 10Gehel: Add types to some fields used by nginx [puppet] - 10https://gerrit.wikimedia.org/r/386317 (https://phabricator.wikimedia.org/T178530) (owner: 10Smalyshev) [08:59:00] (03CR) 10Gehel: [C: 032] Add types to some fields used by nginx [puppet] - 10https://gerrit.wikimedia.org/r/386317 (https://phabricator.wikimedia.org/T178530) (owner: 10Smalyshev) [09:05:17] (03PS2) 10Gehel: wdqs: add timestamp to GC logs [puppet] - 10https://gerrit.wikimedia.org/r/386132 (https://phabricator.wikimedia.org/T175919) [09:05:57] (03CR) 10Gehel: [C: 032] wdqs: add timestamp to GC logs [puppet] - 10https://gerrit.wikimedia.org/r/386132 (https://phabricator.wikimedia.org/T175919) (owner: 10Gehel) [09:07:04] !log rolling restart of all wdqs nodes for GC config change - T175919 [09:07:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:07:11] T175919: investigate GC times on wikidata query service - https://phabricator.wikimedia.org/T175919 [09:07:55] (03PS1) 10Ladsgroup: labs: Whitelist jenkins and Sauce labs IP ranges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386343 (https://phabricator.wikimedia.org/T167432) [09:10:03] (03CR) 10Ladsgroup: [C: 032] labs: Whitelist jenkins and Sauce labs IP ranges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386343 (https://phabricator.wikimedia.org/T167432) (owner: 10Ladsgroup) [09:11:41] (03Merged) 10jenkins-bot: labs: Whitelist jenkins and Sauce labs IP ranges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386343 (https://phabricator.wikimedia.org/T167432) (owner: 10Ladsgroup) [09:11:53] (03CR) 10jenkins-bot: labs: Whitelist jenkins and Sauce labs IP ranges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386343 (https://phabricator.wikimedia.org/T167432) (owner: 10Ladsgroup) [09:14:35] (03PS1) 10Elukey: profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) [09:16:44] (03PS2) 10Elukey: profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) [09:17:39] (03CR) 10jerkins-bot: [V: 04-1] profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [09:19:18] yes jenkins you are right [09:21:28] (03PS3) 10Elukey: profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) [09:24:23] (03PS2) 10Giuseppe Lavagetto: Fix apt_remove [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385957 [09:24:25] (03PS2) 10Giuseppe Lavagetto: Add namespace support [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385969 [09:24:27] (03PS2) 10Giuseppe Lavagetto: Add support for nightly builds [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385970 [09:30:35] (03PS1) 10Filippo Giunchedi: hieradata: extend syslog-tls eqiad rollout [puppet] - 10https://gerrit.wikimedia.org/r/386347 (https://phabricator.wikimedia.org/T136312) [09:31:13] !log upgrade mw1221-mw1235 to wikidiff2 (API servers) [09:31:15] (03PS1) 10Gilles: Upgrade to 1.7 [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/386348 (https://phabricator.wikimedia.org/T178974) [09:31:17] (03CR) 10Elukey: "pcc shows no-op: https://puppet-compiler.wmflabs.org/compiler02/8453/" [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [09:31:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:32:57] (03PS3) 10Giuseppe Lavagetto: Add support for nightly builds [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385970 [09:33:46] (03CR) 10Cparle: [C: 031] Upgrade to 1.7 [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/386348 (https://phabricator.wikimedia.org/T178974) (owner: 10Gilles) [09:35:12] (03CR) 10Giuseppe Lavagetto: [C: 032] Fix apt_remove [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385957 (owner: 10Giuseppe Lavagetto) [09:36:47] (03CR) 10Giuseppe Lavagetto: Add namespace support (031 comment) [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385969 (owner: 10Giuseppe Lavagetto) [09:38:51] (03CR) 10Giuseppe Lavagetto: [C: 032] Add namespace support [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385969 (owner: 10Giuseppe Lavagetto) [09:38:59] (03CR) 10Giuseppe Lavagetto: [C: 032] Add support for nightly builds [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/385970 (owner: 10Giuseppe Lavagetto) [09:39:34] <_joe_> win 19 [09:58:51] (03PS1) 10Marostegui: db-codfw.php: Repool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386351 (https://phabricator.wikimedia.org/T178359) [10:02:12] (03CR) 10Filippo Giunchedi: Upgrade to 1.7 (031 comment) [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/386348 (https://phabricator.wikimedia.org/T178974) (owner: 10Gilles) [10:13:21] 10Operations, 10Epic, 10Goal, 10Services (doing), and 2 others: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. - https://phabricator.wikimedia.org/T169937#3709238 (10mobrovac) [10:13:45] 10Operations, 10Epic, 10Goal, 10Services (doing), and 2 others: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. - https://phabricator.wikimedia.org/T169937#3413824 (10mobrovac) [10:14:39] 10Operations, 10Epic, 10Goal, 10Services (done), and 2 others: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. - https://phabricator.wikimedia.org/T169937#3413824 (10mobrovac) 05Open>03Resolved The objective has been achieved. Resolving. [10:18:40] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [10:19:04] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386353 [10:19:08] (03PS2) 10Marostegui: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386353 [10:30:58] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386353 (owner: 10Marostegui) [10:31:20] !log mobrovac@tin Started deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100 [10:31:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:31:28] T158100: Deprecate and remove the public title/{title} endpoint - https://phabricator.wikimedia.org/T158100 [10:32:10] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386353 (owner: 10Marostegui) [10:32:23] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1077" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386353 (owner: 10Marostegui) [10:32:54] !log upgrade remaining video scalers in eqiad to wikidiff2 1.5.1 [10:33:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:33:15] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s) [10:33:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:33:22] T164488: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488 [10:37:23] (03PS1) 10Marostegui: install_server: Reimage db2091 as stretch [puppet] - 10https://gerrit.wikimedia.org/r/386357 (https://phabricator.wikimedia.org/T170662) [10:38:33] (03CR) 10Marostegui: [C: 032] install_server: Reimage db2091 as stretch [puppet] - 10https://gerrit.wikimedia.org/r/386357 (https://phabricator.wikimedia.org/T170662) (owner: 10Marostegui) [10:40:38] (03PS2) 10Mobrovac: [Beta Labs] Use only EventBus for job processing. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386164 (owner: 10Ppchelko) [10:41:01] !log mobrovac@tin Finished deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100 (duration: 09m 41s) [10:41:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:41:11] T158100: Deprecate and remove the public title/{title} endpoint - https://phabricator.wikimedia.org/T158100 [10:41:15] (03CR) 10Ppchelko: [C: 031] [Beta Labs] Use only EventBus for job processing. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386164 (owner: 10Ppchelko) [10:41:44] (03CR) 10Ppchelko: [C: 031] [Beta Labs] Use only EventBus for job processing. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386164 (owner: 10Ppchelko) [10:42:29] (03CR) 10Mobrovac: [C: 032] [Beta Labs] Use only EventBus for job processing. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386164 (owner: 10Ppchelko) [10:43:36] (03Merged) 10jenkins-bot: [Beta Labs] Use only EventBus for job processing. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386164 (owner: 10Ppchelko) [10:44:11] (03PS1) 10Elukey: site.pp: set db1108 as analyics db replica [puppet] - 10https://gerrit.wikimedia.org/r/386359 (https://phabricator.wikimedia.org/T177405) [10:45:00] !log mobrovac@tin Started scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ [10:45:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:45:19] (03PS1) 10Marostegui: mariadb: Add db2091 to s2 and s4 [puppet] - 10https://gerrit.wikimedia.org/r/386360 (https://phabricator.wikimedia.org/T178359) [10:46:18] (03CR) 10jenkins-bot: [Beta Labs] Use only EventBus for job processing. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386164 (owner: 10Ppchelko) [10:46:44] (03PS1) 10Mobrovac: Revert "[Beta Labs] Use only EventBus for job processing." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386361 [10:47:37] (03CR) 10Marostegui: [C: 032] "Puppet looks good: https://puppet-compiler.wmflabs.org/compiler02/8454/" [puppet] - 10https://gerrit.wikimedia.org/r/386360 (https://phabricator.wikimedia.org/T178359) (owner: 10Marostegui) [10:48:18] (03CR) 10Marostegui: [C: 031] site.pp: set db1108 as analyics db replica [puppet] - 10https://gerrit.wikimedia.org/r/386359 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [10:48:30] (03CR) 10Marostegui: [C: 031] profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [10:48:34] (03CR) 10Elukey: profile::mariadb::misc::eventlogging: add support for systemd (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [10:49:43] !log mobrovac@tin Finished scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ (duration: 04m 43s) [10:49:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:49:50] (03CR) 10Elukey: [C: 032] "Nope my eyes didn't parse correctly, all good, merging!" [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [10:49:55] (03PS4) 10Elukey: profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) [10:50:12] (03CR) 10Elukey: [V: 032 C: 032] profile::mariadb::misc::eventlogging: add support for systemd [puppet] - 10https://gerrit.wikimedia.org/r/386346 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [10:50:28] (03PS1) 10Marostegui: s2,s4.hosts: Add db2091 to s2 and s4 [software] - 10https://gerrit.wikimedia.org/r/386363 (https://phabricator.wikimedia.org/T178359) [10:52:43] (03PS2) 10Elukey: site.pp: set db1108 as analyics db replica [puppet] - 10https://gerrit.wikimedia.org/r/386359 (https://phabricator.wikimedia.org/T177405) [10:53:54] (03CR) 10Marostegui: [C: 032] s2,s4.hosts: Add db2091 to s2 and s4 [software] - 10https://gerrit.wikimedia.org/r/386363 (https://phabricator.wikimedia.org/T178359) (owner: 10Marostegui) [10:55:39] (03Merged) 10jenkins-bot: s2,s4.hosts: Add db2091 to s2 and s4 [software] - 10https://gerrit.wikimedia.org/r/386363 (https://phabricator.wikimedia.org/T178359) (owner: 10Marostegui) [11:17:18] (03PS2) 10Marostegui: db-codfw.php: Repool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386351 (https://phabricator.wikimedia.org/T178359) [11:21:00] (03CR) 10Marostegui: [C: 032] db-codfw.php: Repool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386351 (https://phabricator.wikimedia.org/T178359) (owner: 10Marostegui) [11:21:14] (03PS1) 10JakobVoss: Identify publisher with URI [dumps/dcat] - 10https://gerrit.wikimedia.org/r/386366 [11:22:08] (03Merged) 10jenkins-bot: db-codfw.php: Repool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386351 (https://phabricator.wikimedia.org/T178359) (owner: 10Marostegui) [11:22:17] (03CR) 10jenkins-bot: db-codfw.php: Repool db2038 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386351 (https://phabricator.wikimedia.org/T178359) (owner: 10Marostegui) [11:23:18] !log marostegui@tin Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s) [11:23:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:23:26] T178359: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359 [11:30:44] !log upgrade mw1280-mw1290, mw1312-mw1318 to wikidiff2 1.5.1 [11:30:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:33:10] PROBLEM - puppet last run on cp2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:38:43] (03PS2) 10JakobVoss: Identify publisher with URI [dumps/dcat] - 10https://gerrit.wikimedia.org/r/386366 [11:43:00] Can any root check on terbium whether there have been any recent php killings? [11:45:54] having a look [11:46:20] and maybe also see what cron gives there in hte last hour? [11:47:39] (03CR) 10Elukey: [C: 032] site.pp: set db1108 as analyics db replica [puppet] - 10https://gerrit.wikimedia.org/r/386359 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [11:48:13] hoo: not seeing any segfaults, also no oomkiller (if you meant that) [11:48:22] yeah :/ [11:48:31] Does cron report any non-zero exits recently? [11:48:35] what do you mean with "what cron does", errors from crond itself? [11:48:49] No, I mean non-zero exists, sorry [11:50:26] nothing unusual, but I can also bounce you today's log via mail if you want [11:51:33] Would be appreciated [11:57:02] (03PS1) 10Ppchelko: [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 [11:58:10] RECOVERY - puppet last run on cp2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:58:47] (03CR) 10jerkins-bot: [V: 04-1] [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 (owner: 10Ppchelko) [11:59:11] (03CR) 10Filippo Giunchedi: [C: 032] hieradata: extend syslog-tls eqiad rollout [puppet] - 10https://gerrit.wikimedia.org/r/386347 (https://phabricator.wikimedia.org/T136312) (owner: 10Filippo Giunchedi) [11:59:18] (03PS2) 10Filippo Giunchedi: hieradata: extend syslog-tls eqiad rollout [puppet] - 10https://gerrit.wikimedia.org/r/386347 (https://phabricator.wikimedia.org/T136312) [12:00:35] (03PS2) 10Ppchelko: [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 [12:01:43] (03CR) 10jerkins-bot: [V: 04-1] [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 (owner: 10Ppchelko) [12:02:57] (03PS7) 10Ema: VCL: Exp cache admission policy for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) [12:02:59] (03PS3) 10Ppchelko: [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 [12:06:09] (03PS1) 10Elukey: profile::mariadb::misc::eventlogging:repl: fix systemd template [puppet] - 10https://gerrit.wikimedia.org/r/386370 (https://phabricator.wikimedia.org/T177405) [12:06:45] (03CR) 10Elukey: [C: 032] profile::mariadb::misc::eventlogging:repl: fix systemd template [puppet] - 10https://gerrit.wikimedia.org/r/386370 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [12:08:11] (03PS4) 10Ppchelko: [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 [12:10:44] (03PS1) 10Elukey: profile::mariadb::misc::eventlogging:repl: fix typo in rsyslog config [puppet] - 10https://gerrit.wikimedia.org/r/386371 (https://phabricator.wikimedia.org/T177405) [12:11:48] (03CR) 10Elukey: [C: 032] profile::mariadb::misc::eventlogging:repl: fix typo in rsyslog config [puppet] - 10https://gerrit.wikimedia.org/r/386371 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [12:12:17] !log upgrade mw1266-mw1275 to wikidiff2 1.5.1 [12:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:12:45] (03CR) 10Mobrovac: [C: 032] [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 (owner: 10Ppchelko) [12:14:00] (03Merged) 10jenkins-bot: [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 (owner: 10Ppchelko) [12:14:40] (03PS1) 10Filippo Giunchedi: centralserver: add icinga monitoring for syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386372 (https://phabricator.wikimedia.org/T136312) [12:16:21] (03CR) 10jenkins-bot: [Logging] Enable JobExecutor logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386369 (owner: 10Ppchelko) [12:16:27] !log mobrovac@tin Synchronized wmf-config/InitialiseSettings.php: Activate warning logging for JobExecutor (duration: 00m 50s) [12:16:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:17:37] !log mobrovac@tin Synchronized wmf-config/InitialiseSettings-labs.php: Activate warning logging for JobExecutor (duration: 00m 50s) [12:17:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:40] (03PS2) 10Gilles: Upgrade to 1.7 [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/386348 (https://phabricator.wikimedia.org/T178974) [12:30:02] (03CR) 10Gilles: Upgrade to 1.7 (031 comment) [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/386348 (https://phabricator.wikimedia.org/T178974) (owner: 10Gilles) [12:36:39] !log upgrade mw2163-mw2179 to wikidiff2 1.5.1 [12:36:39] (03PS1) 10Elukey: profile::mariadb::misc::eventlogging::database: support mariadb 10.1 [puppet] - 10https://gerrit.wikimedia.org/r/386376 (https://phabricator.wikimedia.org/T177405) [12:36:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:37:21] (03CR) 10Marostegui: [C: 031] profile::mariadb::misc::eventlogging::database: support mariadb 10.1 [puppet] - 10https://gerrit.wikimedia.org/r/386376 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [12:37:40] (03PS2) 10Filippo Giunchedi: centralserver: add icinga monitoring for syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386372 (https://phabricator.wikimedia.org/T136312) [12:38:56] (03CR) 10jerkins-bot: [V: 04-1] centralserver: add icinga monitoring for syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386372 (https://phabricator.wikimedia.org/T136312) (owner: 10Filippo Giunchedi) [12:45:36] (03CR) 10Elukey: [C: 032] profile::mariadb::misc::eventlogging::database: support mariadb 10.1 [puppet] - 10https://gerrit.wikimedia.org/r/386376 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [12:46:31] (03PS21) 10Paladox: gerrit: Ajust scap files (DO NOT MERGE) [software/gerrit] - 10https://gerrit.wikimedia.org/r/363738 [12:46:33] (03PS19) 10Paladox: Gerrit: Upgrading gerrit to 2.14.6-pre (DO NOT MERGE) [software/gerrit] - 10https://gerrit.wikimedia.org/r/363734 [12:50:04] (03PS1) 10Elukey: profile::mariadb::misc::eventlogging: fix systemd unit template [puppet] - 10https://gerrit.wikimedia.org/r/386379 (https://phabricator.wikimedia.org/T177405) [12:50:59] (03CR) 10Elukey: [C: 032] profile::mariadb::misc::eventlogging: fix systemd unit template [puppet] - 10https://gerrit.wikimedia.org/r/386379 (https://phabricator.wikimedia.org/T177405) (owner: 10Elukey) [12:51:14] (03PS3) 10Filippo Giunchedi: centralserver: add icinga monitoring for syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386372 (https://phabricator.wikimedia.org/T136312) [12:52:05] (03CR) 10Filippo Giunchedi: [C: 032] Upgrade to 1.7 [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/386348 (https://phabricator.wikimedia.org/T178974) (owner: 10Gilles) [12:52:18] (03CR) 10Filippo Giunchedi: [C: 032] centralserver: add icinga monitoring for syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386372 (https://phabricator.wikimedia.org/T136312) (owner: 10Filippo Giunchedi) [12:52:23] (03PS4) 10Filippo Giunchedi: centralserver: add icinga monitoring for syslog-tls [puppet] - 10https://gerrit.wikimedia.org/r/386372 (https://phabricator.wikimedia.org/T136312) [13:00:05] addshore, hashar, anomie, RainbowSprinkles, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor I � Unicode. All rise for European Mid-day SWAT(Max 8 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20171025T1300). [13:00:05] aharoni and Deskana: A patch you scheduled for European Mid-day SWAT(Max 8 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:32] hallo [13:00:37] o/ [13:00:39] (03PS8) 10Ema: VCL: Exp cache admission policy for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) [13:01:00] I'm deploying the Compact Language Links thing, with matt_flaschen [13:01:17] and kart_ is watching us :) [13:02:23] Greetings, all. [13:03:34] Deskana, I'll also run SWAT. [13:04:42] !log About to run ULSCompactLinksDisablePref.php: Setting user preferences for Compact Language Links on dewiki, before removing Beta Feature and changing to regular preference. T177836 [13:04:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:04:50] T177836: Deploy Compact Language Links on the German Wikipedia - https://phabricator.wikimedia.org/T177836 [13:05:37] (03PS5) 10Rush: aborrero: new opsen user and key [puppet] - 10https://gerrit.wikimedia.org/r/385988 (https://phabricator.wikimedia.org/T178807) [13:07:54] matt_flaschen: is it too late to add one more config patch to the window? [13:08:33] 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Requesting access to ops for aborrero - https://phabricator.wikimedia.org/T178809#3709722 (10chasemp) I was cc'd on the go ahead from legal that @aborrero is squared away with them. I'm merging his account now. [13:08:45] ori, no, go ahead. [13:09:08] (03CR) 10Rush: [C: 032] aborrero: new opsen user and key [puppet] - 10https://gerrit.wikimedia.org/r/385988 (https://phabricator.wikimedia.org/T178807) (owner: 10Rush) [13:13:48] 10Operations, 10Pybal, 10Traffic, 10netops, 10Patch-For-Review: Frequent RST returned by appservers to LVS hosts - https://phabricator.wikimedia.org/T163674#3709732 (10elukey) Addendum to the summary: the rst does not happen in case a HTTP connection explicitly asks to be kept alive, since nginx does not... [13:13:50] (03PS1) 10Ori.livneh: Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) [13:15:03] !log merge arturo's key as part of ops [13:15:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:15:44] (03CR) 10Gilles: [C: 031] Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) (owner: 10Ori.livneh) [13:19:09] matt_flaschen: thanks, https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1773578&oldid=1773575 [13:19:51] (ori: just want to say hi and it's great to see you doing ori things) [13:20:26] chasemp: hi!! good to see you too :) [13:22:02] Deskana, starting with the feedback patch now. [13:22:08] Ready! [13:22:49] (03CR) 10Mattflaschen: [C: 032] Updating wikis with consolidate editing feedback [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386222 (https://phabricator.wikimedia.org/T168886) (owner: 10Deskana) [13:22:50] (03PS2) 10Mattflaschen: Updating wikis with consolidate editing feedback [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386222 (https://phabricator.wikimedia.org/T168886) (owner: 10Deskana) [13:23:01] (03CR) 10Mattflaschen: [C: 032] Updating wikis with consolidate editing feedback [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386222 (https://phabricator.wikimedia.org/T168886) (owner: 10Deskana) [13:23:17] (03PS2) 10Elukey: hadoop: raise Xmx/Xms settings for hadoop worker daemons on an1030 [puppet] - 10https://gerrit.wikimedia.org/r/386147 (https://phabricator.wikimedia.org/T178876) [13:24:14] (03Merged) 10jenkins-bot: Updating wikis with consolidate editing feedback [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386222 (https://phabricator.wikimedia.org/T168886) (owner: 10Deskana) [13:24:47] (03PS3) 10Elukey: hadoop: raise Xmx/Xms settings for hadoop worker daemons on an1030 [puppet] - 10https://gerrit.wikimedia.org/r/386147 (https://phabricator.wikimedia.org/T178876) [13:25:06] (03CR) 10Elukey: ">" (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/386147 (https://phabricator.wikimedia.org/T178876) (owner: 10Elukey) [13:26:04] (03PS4) 10Elukey: hadoop: raise Xmx/Xms settings for hadoop worker daemons on an1030 [puppet] - 10https://gerrit.wikimedia.org/r/386147 (https://phabricator.wikimedia.org/T178876) [13:26:06] matt_flaschen: Am I good to test it now? This is my first time being solely responsible for a SWAT deploy, so I don't know the details. [13:26:22] (03CR) 10jenkins-bot: Updating wikis with consolidate editing feedback [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386222 (https://phabricator.wikimedia.org/T168886) (owner: 10Deskana) [13:27:44] (03CR) 10Elukey: [C: 032] hadoop: raise Xmx/Xms settings for hadoop worker daemons on an1030 [puppet] - 10https://gerrit.wikimedia.org/r/386147 (https://phabricator.wikimedia.org/T178876) (owner: 10Elukey) [13:28:04] (03PS1) 10Dmaza: Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) [13:28:26] Deskana, no, I'll let you know. [13:28:34] Thanks. :-) [13:28:50] 10Operations, 10Cloud-Services, 10DBA, 10Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3709749 (10Marostegui) [13:29:23] Deskana, do you know how to test on mwdebug1002? [13:29:37] matt_flaschen: No. [13:30:00] !log restart yarn nodemanager and hdfs datanode on analytics1030 to apply new JVM settings - T178876 [13:30:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:30:08] T178876: Test and possibly raise the Xmx/Xms settings for the Hadoop Yarn Namenode and HDFS datanode daemons - https://phabricator.wikimedia.org/T178876 [13:30:09] (03PS1) 10Rush: puppetmaster: make hiera lookups class params [puppet] - 10https://gerrit.wikimedia.org/r/386386 (https://phabricator.wikimedia.org/T171494) [13:30:23] (03PS2) 10Rush: puppetmaster: make hiera lookups class params [puppet] - 10https://gerrit.wikimedia.org/r/386386 (https://phabricator.wikimedia.org/T171494) [13:30:24] Deskana, please install https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb and then follow https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb . [13:30:26] !log deploy thumbor 1.7 - T178974 [13:30:32] (03PS9) 10Ema: VCL: Exp cache admission policy for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) [13:30:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:30:34] T178974: SVG language URLs don't allow language names with - in them - https://phabricator.wikimedia.org/T178974 [13:30:34] https://wikitech.wikimedia.org/wiki/X-Wikimedia-Debug#Staging_changes [13:30:50] Will do. [13:31:51] matt_flaschen: Okay, so, I got the extension enabled and I've set it to mwdebug1002 and turned it on. Now I just test on, for example, fr.wikipedia.org as normal? [13:32:01] Deskana, correct. [13:32:10] (03PS2) 10Dmaza: Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) [13:33:38] matt_flaschen: Thanks for your patience! The config change works. :-) [13:34:10] Deskana, great. Now I'll deploy it fully, then I'll ask you to retest without mwdebug1002. [13:36:44] !log mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: T168886: Updating wikis with consolidate editing feedback (duration: 00m 51s) [13:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:51] T168886: Change in-editor feedback tool to point to mediawiki.org - https://phabricator.wikimedia.org/T168886 [13:37:14] Deskana, okay, please retest with the extension off. [13:38:07] Hm, that doesn't seem to have worked. [13:38:31] Let me check again. [13:39:37] matt_flaschen: No, it's not working now. [13:40:21] (03CR) 10Rush: "http://puppet-compiler.wmflabs.org/8459/" [puppet] - 10https://gerrit.wikimedia.org/r/386386 (https://phabricator.wikimedia.org/T171494) (owner: 10Rush) [13:40:31] I don't see why it shouldn't be. I tested it in a different browser with a different account too, and it still didn't work. [13:42:22] Hm. I think it's working now. Maybe it just took a moment. [13:43:09] matt_flaschen: Yes, we're all good now. It's working. I'm not sure why it took a little bit. [13:43:59] Deskana, okay, the ResourceLoader module was probably cached. [13:46:51] (03CR) 10Alexandros Kosiaris: [C: 031] "nice!" [puppet] - 10https://gerrit.wikimedia.org/r/386386 (https://phabricator.wikimedia.org/T171494) (owner: 10Rush) [13:46:55] (03PS1) 10ArielGlenn: generate one config fule for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) [13:47:39] (03CR) 10jerkins-bot: [V: 04-1] generate one config fule for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) (owner: 10ArielGlenn) [13:48:05] (03CR) 10Rush: [C: 032] puppetmaster: make hiera lookups class params [puppet] - 10https://gerrit.wikimedia.org/r/386386 (https://phabricator.wikimedia.org/T171494) (owner: 10Rush) [13:49:25] (03PS1) 10ArielGlenn: Make a few more config settings parseable per-project. [dumps] - 10https://gerrit.wikimedia.org/r/386389 (https://phabricator.wikimedia.org/T178893) [13:49:46] (03CR) 10jerkins-bot: [V: 04-1] Make a few more config settings parseable per-project. [dumps] - 10https://gerrit.wikimedia.org/r/386389 (https://phabricator.wikimedia.org/T178893) (owner: 10ArielGlenn) [13:55:33] (03CR) 10Mattflaschen: [C: 032] Deploy Compact Language Links on the German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384527 (https://phabricator.wikimedia.org/T177836) (owner: 10KartikMistry) [13:56:01] PROBLEM - rsyslog TLS listener on port 6514 on wezen is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer [13:56:12] (03CR) 10BBlack: [C: 04-1] "Looking good! Various inline commentary follows!" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) (owner: 10Ema) [13:56:12] RECOVERY - rsyslog TLS listener on port 6514 on wezen is OK: SSL OK - Certificate wezen.codfw.wmnet valid until 2021-08-21 20:09:05 +0000 (expires in 1396 days) [14:01:50] (03PS2) 10ArielGlenn: Make a few more config settings parseable per-project. [dumps] - 10https://gerrit.wikimedia.org/r/386389 (https://phabricator.wikimedia.org/T178893) [14:02:40] 10Operations, 10Traffic, 10Wikimedia-Logstash: Varnish does not vary elasticsearch query by request body - https://phabricator.wikimedia.org/T174960#3709805 (10dbarratt) I tried a `POST` request to the `_msearch` endpoint, but got this response: ``` { "statusCode": 400, "error": "Bad Request", "message":... [14:04:00] (03CR) 10Dbarratt: [C: 031] Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) (owner: 10Dmaza) [14:04:31] PROBLEM - rsyslog TLS listener on port 6514 on wezen is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer [14:04:52] RECOVERY - rsyslog TLS listener on port 6514 on wezen is OK: SSL OK - Certificate wezen.codfw.wmnet valid until 2021-08-21 20:09:05 +0000 (expires in 1396 days) [14:05:08] (03PS2) 10ArielGlenn: generate one config fule for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) [14:05:27] Deskana, please test the other one on mwdebug1002. [14:05:41] (03CR) 10jerkins-bot: [V: 04-1] generate one config fule for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) (owner: 10ArielGlenn) [14:05:50] 10Operations, 10Pybal, 10Traffic, 10netops, 10Patch-For-Review: Frequent RST returned by appservers to LVS hosts - https://phabricator.wikimedia.org/T163674#3709808 (10BBlack) > would it mean that nginx should keep more TCP connections opened hoping for the client to eventually send a close notify (in re... [14:07:28] matt_flaschen: Confirmed working. [14:08:29] 10Operations, 10Traffic, 10Wikimedia-Logstash: Varnish does not vary elasticsearch query by request body - https://phabricator.wikimedia.org/T174960#3578041 (10dcausse) Yes the syntax is slightly different: - you need to set `Content-Type: application/x-ndjson` - every request must be formed of 2 lines: -- f... [14:10:03] !log mattflaschen@tin Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: (no justification provided) (duration: 00m 50s) [14:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:11:13] !log 'Deployed T178933: MWInternalLinkContextItem: increase specificity to override OOUI changes' [14:11:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:11:21] T178933: Images shown in the link tool are very short and wide at cswiki - https://phabricator.wikimedia.org/T178933 [14:12:54] We're over time, but I'll keep going, since nothing is after. [14:13:10] thanks [14:13:56] Deskana, it's deployed now, so please re-test without mwdebug1002. Might take a couple more minutes due to caching. [14:16:37] matt_flaschen: Looks good! Thanks! [14:17:21] Deskana, great, thanks. [14:22:07] (03CR) 10Chad: git-sync-upstream: rewrite in python (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [14:22:18] dcausse, please test both wmf.4 and wmf.5 on mwdebug1002. [14:22:26] matt_flaschen: looking [14:24:10] 10Operations, 10Traffic, 10Wikimedia-Logstash: Varnish does not vary elasticsearch query by request body - https://phabricator.wikimedia.org/T174960#3709846 (10dbarratt) >>! In T174960#3709823, @dcausse wrote: > -- first line some metadata such as the index you want to query What is our index? errr.. what i... [14:24:28] (03CR) 10Andrew Bogott: "right as always" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [14:24:38] (03PS5) 10Andrew Bogott: git-sync-upstream: rewrite in python [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) [14:24:40] (03PS6) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [14:25:09] godog: if you don't midn then [14:25:11] i'll merge https://gerrit.wikimedia.org/r/#/c/386190/ [14:25:17] and add some customizations for mirror maker [14:25:32] !log moved jmxtrans and prometheus-jmx-exporter from thirdparty to main as part of repo reorg for stretch-wikimedia [14:25:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:26:26] (03PS1) 10Chad: group1 to wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386392 [14:26:28] (03CR) 10Chad: [C: 04-2] group1 to wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386392 (owner: 10Chad) [14:27:55] ottomata: sec [14:28:03] (03PS1) 10BBlack: Merge branch 'wmf-1.13' into wmf-1.13-jessie [software/nginx] (wmf-1.13-jessie) - 10https://gerrit.wikimedia.org/r/386393 [14:28:12] (03CR) 10Mattflaschen: [C: 032] Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) (owner: 10Ori.livneh) [14:29:10] (03PS2) 10Mattflaschen: Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) (owner: 10Ori.livneh) [14:29:11] (03PS1) 10Filippo Giunchedi: Revert "hieradata: extend syslog-tls eqiad rollout" [puppet] - 10https://gerrit.wikimedia.org/r/386394 [14:29:37] (03CR) 10Mattflaschen: [C: 032] Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) (owner: 10Ori.livneh) [14:30:00] matt_flaschen: all good [14:30:27] dcausse, thanks, let me deploy them now. [14:30:51] (03PS3) 10ArielGlenn: generate one config fule for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) [14:30:53] (03Merged) 10jenkins-bot: Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) (owner: 10Ori.livneh) [14:31:09] (03CR) 10jenkins-bot: Set $wgAutoloadAttemptLowercase to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386384 (https://phabricator.wikimedia.org/T166759) (owner: 10Ori.livneh) [14:35:56] (03CR) 10Filippo Giunchedi: [C: 032] Revert "hieradata: extend syslog-tls eqiad rollout" [puppet] - 10https://gerrit.wikimedia.org/r/386394 (owner: 10Filippo Giunchedi) [14:36:13] !log mattflaschen@tin Synchronized php-1.31.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s) [14:36:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:36:21] T177956: Turn off: A/B test to test relaxing the retrieval query filter - https://phabricator.wikimedia.org/T177956 [14:37:41] !log mattflaschen@tin Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s) [14:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:38:02] dcausse, deployed. Please test without mwdebug1002. [14:40:24] (03PS1) 10Muehlenhoff: Use thirdparty/k8s repository in docker class [puppet] - 10https://gerrit.wikimedia.org/r/386395 [14:40:50] (03CR) 10jerkins-bot: [V: 04-1] Use thirdparty/k8s repository in docker class [puppet] - 10https://gerrit.wikimedia.org/r/386395 (owner: 10Muehlenhoff) [14:41:28] 10Operations, 10Traffic, 10Wikimedia-Logstash: Varnish does not vary elasticsearch query by request body - https://phabricator.wikimedia.org/T174960#3709879 (10dcausse) @dbarratt sadly I don't know all the details of this cluster, but you could get it working by not specifying an index: with a `requests` f... [14:41:35] (03PS2) 10Muehlenhoff: Use thirdparty/k8s repository in docker class [puppet] - 10https://gerrit.wikimedia.org/r/386395 [14:42:03] matt_flaschen: tested and works as expected, thanks for the deploy! [14:42:08] ori, patch is on mwdebug1002. [14:42:09] (03Abandoned) 10Gehel: ocg: switch to LVS endpoint for logstash [puppet] - 10https://gerrit.wikimedia.org/r/380995 (https://phabricator.wikimedia.org/T175242) (owner: 10Gehel) [14:42:10] Thanks, dcausse [14:43:12] matt_flaschen: looks good [14:47:09] !log mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: T166759: Set $wgAutoloadAttemptLowercase to false (duration: 00m 50s) [14:47:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:16] T166759: If possible, turn off $wgAutoloadAttemptLowercase - https://phabricator.wikimedia.org/T166759 [14:47:56] \o/ [14:48:00] thanks matt_flaschen! [14:48:53] ori, please test without mwdebug1002. [14:49:13] (03PS2) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [14:49:56] (03PS4) 10Mattflaschen: Deploy Compact Language Links on the German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384527 (https://phabricator.wikimedia.org/T177836) (owner: 10KartikMistry) [14:50:06] (03CR) 10Mattflaschen: [C: 032] Deploy Compact Language Links on the German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384527 (https://phabricator.wikimedia.org/T177836) (owner: 10KartikMistry) [14:50:13] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [14:51:30] !log update logstash template on logstash elsaticsearch cluster for T178530 [14:51:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:51:37] T178530: Improve field mapping for nginx logstash - https://phabricator.wikimedia.org/T178530 [14:51:45] (03Merged) 10jenkins-bot: Deploy Compact Language Links on the German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384527 (https://phabricator.wikimedia.org/T177836) (owner: 10KartikMistry) [14:51:57] (03CR) 10jenkins-bot: Deploy Compact Language Links on the German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384527 (https://phabricator.wikimedia.org/T177836) (owner: 10KartikMistry) [14:52:33] !log applying new template to elasticsearch / logstash - T178530 [14:52:35] ottomata: tbh I'm more of a fan of explicit vs implicit, if you are going to tweak the config anyways then might as well ship the tweaked config [14:52:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:52:54] (03PS3) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [14:52:56] (03PS4) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [14:53:28] matt_flaschen: thanks again [14:54:46] I did see this fatal in the logs: Notice: Undefined index: 1 in /srv/mediawiki/php-1.31.0-wmf.4/includes/media/FormatMetadata.php on line 744 [14:54:49] about 70/hr [14:55:02] it did not start recently [14:55:06] but still, someone should look [14:55:37] (but it is not associated with this deployment window) [14:55:58] godog: ya, for this one thing i will, but the patch I added woudl be potentially used for others [14:56:01] ori, thanks, I'll make sure it's tracked before I leave. [14:56:05] i guess if you always want explicti then there's no reason to have it [14:57:22] ottomata: yeah IMHO explicit is better [14:57:29] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [14:57:47] yaaa, but wouldn't it be SO much better to not have to configure metrics godog? [14:57:47] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3709938 (10kaythaney) p:05Normal>03High Hi all, We're in the final push to get the site live with Wordpress VIP, and ready to address the DNS issue. We're looking to push this live in the next... [14:57:48] i want magic [14:57:53] i want to say: gimme metrics! [14:57:54] and just get them [14:58:10] reading all these metrics ahead of time for every new service is just so cumbersome! :) [14:58:25] 10Operations, 10Traffic, 10Wikimedia-Logstash: Varnish does not vary elasticsearch query by request body - https://phabricator.wikimedia.org/T174960#3709944 (10dbarratt) @dcausse so that does work, so @ema this is a valid work around, although, imho, it's not elegant. I've [[ https://wikitech.wikimedia.org/... [14:58:26] buuuuut k i see what you saying. will abandon this patch [14:58:51] (03Abandoned) 10Ottomata: Add default_prometheus_jmx_exporter.yaml [puppet] - 10https://gerrit.wikimedia.org/r/386190 (https://phabricator.wikimedia.org/T175344) (owner: 10Ottomata) [14:58:53] (03PS5) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [14:59:09] ottomata: heheh in the magic case you can still have the default config in puppet and call the class with source => the default [14:59:33] oh but then you have to commit the default config to puppet, no? [14:59:56] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:00:16] ottomata: yeah [15:00:38] ori: matt_flaschen - https://phabricator.wikimedia.org/T179004 [15:00:39] (03PS4) 10ArielGlenn: generate one config fule for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) [15:00:51] ori, aharoni filed it. Thanks. [15:00:53] ori: o/ [15:01:05] thanks aharoni. hi elukey! [15:01:31] * godog waves at ori [15:01:36] yo yo [15:02:14] (03PS6) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [15:03:15] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:03:28] (03PS3) 10Volans: Backends: add support to external backends plugins [software/cumin] - 10https://gerrit.wikimedia.org/r/384616 (https://phabricator.wikimedia.org/T178342) [15:03:30] (03PS1) 10Volans: Logging: uniform loggers [software/cumin] - 10https://gerrit.wikimedia.org/r/386399 (https://phabricator.wikimedia.org/T179002) [15:03:32] (03PS1) 10Volans: Logging: use % syntax for parameters [software/cumin] - 10https://gerrit.wikimedia.org/r/386400 (https://phabricator.wikimedia.org/T179002) [15:04:42] 10Operations, 10ops-esams: Degraded RAID on lvs3001 - https://phabricator.wikimedia.org/T168619#3709981 (10Volans) [15:04:44] 10Operations, 10ops-esams: Degraded RAID on lvs3001 - https://phabricator.wikimedia.org/T177881#3709979 (10Volans) [15:05:29] 10Operations, 10ops-esams: Degraded RAID on bast3002 - https://phabricator.wikimedia.org/T177875#3709983 (10Volans) [15:05:31] 10Operations, 10ops-esams: bast3002 sdb broken - https://phabricator.wikimedia.org/T169035#3709985 (10Volans) [15:06:10] aharoni tested, and it works on mwdebug1002. [15:06:21] PROBLEM - rsyslog TLS listener on port 6514 on wezen is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer [15:06:35] godog: ^^^ [15:09:00] (03PS7) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [15:09:06] !log mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: T177836: Deploy Compact Language Links on the German Wikipedia (duration: 00m 49s) [15:09:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:09:14] T177836: Deploy Compact Language Links on the German Wikipedia - https://phabricator.wikimedia.org/T177836 [15:09:48] aharoni, deployed. Please test. [15:10:00] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:10:21] volans: on it [15:10:35] let me know if I can help ;) [15:10:48] as in, expected [15:10:50] I will! [15:10:58] I saw you logged in and didn't look further [15:11:14] (03PS1) 10Filippo Giunchedi: rsyslog: bump maximum fds for rsyslog-receiver [puppet] - 10https://gerrit.wikimedia.org/r/386404 (https://phabricator.wikimedia.org/T136312) [15:11:26] yeah that's ^ the testing [15:11:53] anyways I'll update the task too, doesn't look so good so far even with the fd limit bumped [15:12:10] what's the issue? [15:12:15] doesn't scale? :D [15:12:22] (03Abandoned) 10BBlack: Merge branch 'wmf-1.13' into wmf-1.13-jessie [software/nginx] (wmf-1.13-jessie) - 10https://gerrit.wikimedia.org/r/386393 (owner: 10BBlack) [15:12:29] sigabrt, there's core dumps in /var/tmp/core [15:12:38] (03PS1) 10BBlack: Merge branch 'wmf-1.13' into wmf-1.13-jessie [software/nginx] (wmf-1.13-jessie) - 10https://gerrit.wikimedia.org/r/386405 [15:13:55] (03PS8) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [15:15:03] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:17:02] (03CR) 10Filippo Giunchedi: "PCC https://puppet-compiler.wmflabs.org/compiler02/8463" [puppet] - 10https://gerrit.wikimedia.org/r/386404 (https://phabricator.wikimedia.org/T136312) (owner: 10Filippo Giunchedi) [15:18:25] (03PS9) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [15:19:22] (03CR) 10Volans: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/386404 (https://phabricator.wikimedia.org/T136312) (owner: 10Filippo Giunchedi) [15:19:55] (03PS2) 10Krinkle: Update dumps archive_index.html for the files I just uploaded [puppet] - 10https://gerrit.wikimedia.org/r/383958 (owner: 10Tim Starling) [15:20:11] (03CR) 10Krinkle: [C: 031] "Fixed a spurious slash detected by Gerrit's highlighter." [puppet] - 10https://gerrit.wikimedia.org/r/383958 (owner: 10Tim Starling) [15:21:12] PROBLEM - rsyslog TLS listener on port 6514 on wezen is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer [15:21:31] RECOVERY - rsyslog TLS listener on port 6514 on wezen is OK: SSL OK - Certificate wezen.codfw.wmnet valid until 2021-08-21 20:09:05 +0000 (expires in 1396 days) [15:21:31] !log bounce rsyslog on wezen [15:21:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:26:47] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:27:21] (03PS1) 10Ladsgroup: Make disabled usage aspect use the new config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386408 (https://phabricator.wikimedia.org/T172914) [15:27:50] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe, 10cloud-services-team (FY2017-18): Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3710179 (10Andrew) [15:29:21] PROBLEM - rsyslog TLS listener on port 6514 on wezen is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer [15:30:22] that's me again ^ [15:30:31] RECOVERY - rsyslog TLS listener on port 6514 on wezen is OK: SSL OK - Certificate wezen.codfw.wmnet valid until 2021-08-21 20:09:05 +0000 (expires in 1396 days) [15:30:56] mutante, looks like the deb-upload script needs to be updated to use the new servers then? [15:31:12] !log Power off db1101 for HW maintenance - T178383 [15:31:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:31:25] T178383: db1101 crashed - memory errors - https://phabricator.wikimedia.org/T178383 [15:34:43] (03PS10) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [15:35:51] 10Operations, 10netops, 10Patch-For-Review: Find a new PIM RP IP - https://phabricator.wikimedia.org/T167842#3710211 (10ayounsi) Indeed, confirmed. Thanks! [15:37:48] (03PS5) 10ArielGlenn: generate one config file for xml/sql dumps for wikis [puppet] - 10https://gerrit.wikimedia.org/r/386388 (https://phabricator.wikimedia.org/T178893) [15:39:35] (03PS2) 10Filippo Giunchedi: rsyslog: bump maximum fds for rsyslog-receiver [puppet] - 10https://gerrit.wikimedia.org/r/386404 (https://phabricator.wikimedia.org/T136312) [15:39:59] !log Optimize pagelinks and templatelinks on labsdb1011 - T174509 [15:40:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:40:07] T174509: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509 [15:40:44] (03CR) 10Filippo Giunchedi: [C: 032] rsyslog: bump maximum fds for rsyslog-receiver [puppet] - 10https://gerrit.wikimedia.org/r/386404 (https://phabricator.wikimedia.org/T136312) (owner: 10Filippo Giunchedi) [15:42:31] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:44:16] (03PS11) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [15:45:33] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [15:51:35] (03CR) 10BBlack: [C: 04-1] "Reviewing all of this some more (it's been a while):" [puppet] - 10https://gerrit.wikimedia.org/r/335232 (owner: 10BBlack) [15:56:44] 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Encrypt syslog traffic - https://phabricator.wikimedia.org/T136312#3710273 (10fgiunchedi) Results of today's experiments: initially we were bumping into max open files limit (fixed) and after that was fixed rsyslog was regularly crashing wi... [16:01:52] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3710321 (10Dzahn) Hi @kaythaney i think there is a misunderstanding. There is no wikimediaendowment.org zone file on our side. The domain has never existed in our DNS configuration. And since it's... [16:03:23] !log copied arcconf, hp-health, hpacucli, hpssa, hpssacli, hpssaducli, lsiutil, megacli, megaclisas-status to stretch-wikimedia/thirdparty/hwraid (thirdparty will be dropped at a later point) [16:03:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:05:23] !log upgrade mw2180-mw2189 to wikidiff2 1.5.1 [16:05:28] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3710338 (10Dzahn) regarding the ticket link, i was able to create an account, but after that i just get: ``` This support portal was recently redesigned. The page you were looking for doesn't exist... [16:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:14] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3710355 (10Dzahn) I can see other "organization requests" created by Wikimedia in the past at https://wordpressvip.zendesk.com/hc/en-us/requests/organization but ticket 60140 is not among them. [16:08:03] (03PS10) 10Ema: VCL: Exp cache admission policy for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) [16:08:24] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3710363 (10Dzahn) @kaythaney Either way, i don't think there is anything you need from Operations to move forward on this. There is nothing to configure on our side. [16:09:34] PROBLEM - puppet last run on wezen is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[rsyslog] [16:11:40] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3710373 (10Dzahn) I think the only thing that needs to happen is that legal (or whoever bought the domain name with MarkMonitor) contacts MarkMonitor and tells them the name servers need to be change... [16:12:35] (03CR) 10Ema: "TODO: auto-calculate admission_param and use a boolean flag to toggle the feature instead, understand coalescing behaviors." (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) (owner: 10Ema) [16:14:16] (03CR) 10BBlack: VCL: Exp cache admission policy for varnish-fe (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/386192 (https://phabricator.wikimedia.org/T144187) (owner: 10Ema) [16:22:34] (03CR) 10BBlack: [C: 031] cache: set timeout_idle on text and upload [puppet] - 10https://gerrit.wikimedia.org/r/385985 (https://phabricator.wikimedia.org/T159429) (owner: 10Ema) [16:28:40] (03CR) 10Dzahn: "why don't we want to use the LDAP password and keep a separate password?" [puppet] - 10https://gerrit.wikimedia.org/r/350484 (owner: 10Paladox) [16:29:47] (03CR) 10Paladox: "> why don't we want to use the LDAP password and keep a separate" [puppet] - 10https://gerrit.wikimedia.org/r/350484 (owner: 10Paladox) [16:30:17] 10Operations, 10Patch-For-Review: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735#3710471 (10kaythaney) Thanks, Daniel -- and my apologies for the hassle. Let me loop in our Wordpress contacts. - KT [16:31:11] (03CR) 10Chad: [C: 031] "It was brought up because it puts that same user/password in labs, IIRC." [puppet] - 10https://gerrit.wikimedia.org/r/350484 (owner: 10Paladox) [16:34:31] RECOVERY - puppet last run on wezen is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:35:40] note that ^^, you will have to remeber your password. As it will know longer show it. It's now hashed in gerrit 2.14+ [16:36:30] (03CR) 10Hoo man: [C: 031] "Looks good." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386408 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [16:38:26] !log gerrit: purging last remnants of debian package and running puppet. service may briefly restart [16:38:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:43:32] (03PS12) 10Mobrovac: [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) [16:45:26] (03PS1) 10Hoo man: Enable fine grained usage tracking on statement usage wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386421 (https://phabricator.wikimedia.org/T172914) [16:48:23] 10Operations, 10ops-eqiad, 10DC-Ops, 10cloud-services-team (Kanban): labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3710500 (10Cmjohnson) Dell declined the new system board. We are getting another CPU to since that is the part that seems to be broken. [16:49:18] 10Operations, 10ops-eqiad, 10DBA: db1101 crashed - memory errors - https://phabricator.wikimedia.org/T178383#3710502 (10Cmjohnson) Dell declined to send the new DIMM, stated that my supporting documentation was insufficient. I swapped the DIMM at A4 to B4 and will need to wait for that to fail before submit... [16:51:43] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Improve the checking procedure and emit better messages [software/service-checker] - 10https://gerrit.wikimedia.org/r/386116 (https://phabricator.wikimedia.org/T150560) (owner: 10Mobrovac) [16:59:50] !log upgrade mw2190-mw2199 to wikidiff2 1.5.1 [16:59:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:00:28] 10Operations, 10ops-eqiad, 10DBA: db1101 crashed - memory errors - https://phabricator.wikimedia.org/T178383#3710517 (10Marostegui) How's that possible? Aren't the logs from _their_ server's idrac enough?? [17:03:33] 10Operations, 10ops-eqiad, 10DBA: db1101 crashed - memory errors - https://phabricator.wikimedia.org/T178383#3710521 (10Marostegui) 05Open>03Resolved Anyways, I will mark this as resolved (mysql is back up) and let's reopen once it fails again. I will do some heavy alters in the next few days so we will... [17:10:26] no_justification: for T157414 re dropping from repo, just gerrit 2.13.8+git1-wmf.7 or also some other package? [17:10:27] T157414: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414 [17:10:29] (03PS1) 10BBlack: tlsproxy: use light variant [puppet] - 10https://gerrit.wikimedia.org/r/386424 (https://phabricator.wikimedia.org/T164456) [17:10:43] moritzm: Any versions of that package [17:10:53] For any distros [17:11:44] no_justification: done, removed from jessie and stretch [17:11:51] Sweeeeet [17:11:53] :) [17:12:15] (03CR) 10BBlack: [V: 031] "This compiles, and "works" in the sense that it can be turned on and doesn't completely break everything." [software/nginx] (wmf-1.13) - 10https://gerrit.wikimedia.org/r/386195 (https://phabricator.wikimedia.org/T163674) (owner: 10BBlack) [17:12:22] moritzm: It's scap3 deployed now. Gives us far more flexibility like rolling out a new plugin without rebuilding a huge debian package [17:12:26] I've also doublechecked via cumin that the deb is no longer installed anywhere in prod [17:12:47] no_justification: sure, I've been following the Phab task :-) [17:12:50] :D [17:12:53] Sooooo glad this is done [17:13:09] (03CR) 10BBlack: [V: 031 C: 032] new patch: configurable ssl_do_wait_shutdown [software/nginx] (wmf-1.13) - 10https://gerrit.wikimedia.org/r/386195 (https://phabricator.wikimedia.org/T163674) (owner: 10BBlack) [17:13:13] (03CR) 10BBlack: [C: 032] Release 1.13.6-2+wmf1 for stretch [software/nginx] (wmf-1.13) - 10https://gerrit.wikimedia.org/r/386196 (owner: 10BBlack) [17:13:19] (03CR) 10BBlack: [C: 032] Merge branch 'wmf-1.13' into wmf-1.13-jessie [software/nginx] (wmf-1.13-jessie) - 10https://gerrit.wikimedia.org/r/386405 (owner: 10BBlack) [17:13:44] :) [17:17:28] !log upgrade mw2224-mw2242 to wikidiff2 1.5.1 [17:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:20:44] anyone else no longer seeing download links in the Gerrit interface? [17:20:54] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519#3710582 (10elukey) After re-reading the task mw1329-48 are the only hosts left (20), that should all be in Row C as far as I get, in the following config: * 4... [17:22:30] dbrant hi, what links? [17:22:38] 10Operations, 10Gerrit, 10Patch-For-Review, 10Release-Engineering-Team (Backlog): Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3710584 (10demon) This isn't really stalled, the host has indeed been reimaged as stretch and is completely working. The remaining issue is tracked in... [17:22:45] 10Operations, 10Gerrit: Upload gerrit package to stretch apt.wm.org repo - https://phabricator.wikimedia.org/T165620#3710586 (10demon) [17:22:45] oh [17:22:47] 10Operations, 10Gerrit, 10Patch-For-Review, 10Release-Engineering-Team (Backlog): Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3710585 (10demon) 05stalled>03Resolved [17:22:48] i see [17:22:53] yep [17:22:54] no_justification ^^ [17:22:59] PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 20 seconds [17:23:02] clone links are not showing [17:23:27] Well that's no good. [17:23:52] 10Operations, 10ops-eqiad, 10DBA: db1101 crashed - memory errors - https://phabricator.wikimedia.org/T178383#3710589 (10Marostegui) I have started 16 (1 per database present ) concurrent alters for the templatelinks table to generate some load [17:24:01] does it have the download-command plugin installed? [17:24:09] Um [17:24:13] None of the plugins are installed other than deleteproject. [17:24:16] * no_justification looks [17:24:26] Ohhhh, I bet I know [17:25:10] ah [17:25:28] yeh it's because we purged the gerrit package so we need to redeploy the plugins with scap [17:25:37] RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 579 bytes in 0.019 second response time [17:26:16] Weird that gerrit2001 was ok. [17:26:20] Whatevs [17:26:41] gerrit should refresh plugin status every ~5m so we'll just hang tight [17:26:46] And they're back [17:27:03] alright! [17:27:11] Thanks for reporting, fallout from some cleanup [17:27:16] Easily fixed :) [17:27:20] thx [17:27:34] I still don't see it back [17:27:43] I do. Give it a second, cached? [17:27:44] * no_justification shrugs [17:28:07] interesting, on an incognito tab I can see them back [17:28:31] in the one where I'm logged in not yet, but yeah seems fixed [17:29:10] forcing the disable cache with the developer tools they are back no_justification ;) [17:29:28] Yay [17:34:46] (03PS1) 10Ottomata: Add LVS IP for druid-public-overlord [dns] - 10https://gerrit.wikimedia.org/r/386426 [17:36:59] (03CR) 10Ottomata: [C: 032] Add LVS IP for druid-public-overlord [dns] - 10https://gerrit.wikimedia.org/r/386426 (owner: 10Ottomata) [17:37:30] 10Operations: Backport firejail 0.9.52 for use on Wikimedia appservers - https://phabricator.wikimedia.org/T179022#3710622 (10Legoktm) [17:37:48] 10Operations: Backport firejail 0.9.52 for use on Wikimedia appservers - https://phabricator.wikimedia.org/T179022#3710622 (10Legoktm) 05Open>03stalled Marking as stalled until 0.9.52 is actually released. [17:38:08] PROBLEM - IPMI Sensor Status on analytics1037 is CRITICAL: Sensor Type(s) Temperature, Power_Supply Status: Critical [PS Redundancy = Critical, Status = Critical, Status = Critical] [17:38:47] (03PS1) 10Ottomata: Add LVS for druid-public-overlord indexing service [puppet] - 10https://gerrit.wikimedia.org/r/386427 (https://phabricator.wikimedia.org/T176223) [17:40:55] (03PS2) 10Ottomata: Add LVS for druid-public-overlord indexing service [puppet] - 10https://gerrit.wikimedia.org/r/386427 (https://phabricator.wikimedia.org/T176223) [17:43:13] ottomata: we probably need to mention that the overlord port is not reachable by domain_networks (but it is firewalled) in the lvs config [17:43:49] ah, k adding comment [17:44:15] also is pybal going to be able to do health checks on overlord's port? [17:45:34] the check_http_lvs i think is done by icinga. so probably? is icinga host allowed on all ports? HMm maybe not. [17:45:43] but for pybal availability checks, hm. [17:45:48] i guess pybal would ahve to do them [17:46:43] 10_monitoring-all: saddr $MONITORING_HOSTS ACCEPT; [17:46:47] so that's good. for icinga check [17:47:13] (going to comment in #security to avoid double posting) [17:47:15] :) [17:48:07] (03Abandoned) 10Chad: Stop loading FundraiserLandingPage in beta for now [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386251 (owner: 10Chad) [17:55:09] (03PS6) 10Andrew Bogott: git-sync-upstream: rewrite in python [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) [17:55:11] (03PS7) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [17:56:31] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe, 10cloud-services-team (FY2017-18): Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3710655 (10herron) [18:00:04] addshore, hashar, anomie, RainbowSprinkles, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Morning SWAT (Max 8 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20171025T1800). [18:00:04] Smalyshev, davidwbarratt, DMaza, DMaza, and Amir1: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:00:13] hello [18:00:31] here! [18:00:40] here too [18:01:03] 10Operations, 10Puppet, 10User-Joe: Puppet: Use of 'import' has been discontinued in favor of a manifest directory. - https://phabricator.wikimedia.org/T179023#3710660 (10herron) [18:01:34] I'll swat. [18:01:53] totally misread that as manifest destiny [18:02:53] 10Operations, 10Traffic: Migrate to nginx-light - https://phabricator.wikimedia.org/T164456#3710675 (10BBlack) Seems the bot missed logging this here: https://gerrit.wikimedia.org/r/#/c/386424/ [18:04:05] (03PS2) 10Niharika29: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386231 (https://phabricator.wikimedia.org/T177319) (owner: 10Dbarratt) [18:04:11] (03CR) 10Niharika29: [C: 032] Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386231 (https://phabricator.wikimedia.org/T177319) (owner: 10Dbarratt) [18:05:10] here [18:05:57] (03Merged) 10jenkins-bot: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386231 (https://phabricator.wikimedia.org/T177319) (owner: 10Dbarratt) [18:06:19] (03CR) 10jenkins-bot: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386231 (https://phabricator.wikimedia.org/T177319) (owner: 10Dbarratt) [18:07:24] davidwbarratt: ^ is live on mwdebug1002. [18:07:37] Niharika thanks, testing [18:08:14] Niharika looks good to me! [18:09:31] Amir1: https://gerrit.wikimedia.org/r/#/c/386403/ is live on mwdebug1002. [18:09:41] davidwbarratt: Ack. Syncing. [18:09:53] Niharika: the wmf.5 version is not testable [18:10:02] because no wiki in wmf.5 has ores enabled [18:11:07] Amir1: Alright. I'll pull the wmf.4 one and merge them both once you test it. [18:11:23] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - https://gerrit.wikimedia.org/r/386231 (duration: 00m 51s) [18:11:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:11:30] davidwbarratt: ^ Synced. [18:11:36] Niharika YAY! [18:11:40] 10Operations, 10ops-eqiad, 10hardware-requests, 10User-fgiunchedi: Decommission ms-be1001 - ms-be1012 - https://phabricator.wikimedia.org/T166489#3710701 (10Cmjohnson) [18:11:41] great thanks :) [18:12:04] 10Operations, 10ops-eqiad, 10hardware-requests, 10User-fgiunchedi: Decommission ms-be1001 - ms-be1012 - https://phabricator.wikimedia.org/T166489#3297629 (10Cmjohnson) 05Open>03Resolved resolved [18:12:22] (03PS3) 10Niharika29: Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) (owner: 10Dmaza) [18:12:26] (03CR) 10Niharika29: [C: 032] Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) (owner: 10Dmaza) [18:13:25] 10Operations, 10Puppet, 10User-Joe: Puppet: Use of 'import' has been discontinued in favor of a manifest directory. - https://phabricator.wikimedia.org/T179023#3710709 (10herron) [18:13:30] Niharika: also https://gerrit.wikimedia.org/r/#/c/379426/ :) (just checking it wasn't missed) [18:13:50] SMalyshev: Ah, no it wasn't missed. :) [18:13:56] ok, cool :) [18:14:00] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe, 10cloud-services-team (FY2017-18): Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3710710 (10herron) [18:15:03] (03PS4) 10Niharika29: Make using CirrusSearch engine default for wbsearchentities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/379426 (https://phabricator.wikimedia.org/T175741) (owner: 10Smalyshev) [18:15:12] (03CR) 10Niharika29: [C: 032] "SWAT." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/379426 (https://phabricator.wikimedia.org/T175741) (owner: 10Smalyshev) [18:15:41] (03Merged) 10jenkins-bot: Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) (owner: 10Dmaza) [18:16:20] (03CR) 10jenkins-bot: Add AbuseFilterSlow channel to monolog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386385 (https://phabricator.wikimedia.org/T178853) (owner: 10Dmaza) [18:16:29] Amir1: https://gerrit.wikimedia.org/r/#/c/386402/ is on mwdebug1002 now. [18:16:49] Niharika: works like a charm [18:16:54] thank you :) [18:17:08] (03Merged) 10jenkins-bot: Make using CirrusSearch engine default for wbsearchentities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/379426 (https://phabricator.wikimedia.org/T175741) (owner: 10Smalyshev) [18:17:20] Amir1: Perfect. I'll sync it and the wmf.5 one too. [18:17:28] DMaza: https://gerrit.wikimedia.org/r/#/c/386385/3 is on mwdebug1002 now. [18:17:35] checking [18:18:43] Niharika: the other patches are not testable [18:18:52] (03CR) 10jenkins-bot: Make using CirrusSearch engine default for wbsearchentities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/379426 (https://phabricator.wikimedia.org/T175741) (owner: 10Smalyshev) [18:19:10] !log niharika29@tin Synchronized php-1.31.0-wmf.4/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 51s) [18:19:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:19:19] T178962: "No model available for [models]" error for API access - https://phabricator.wikimedia.org/T178962 [18:19:44] Amir1: Okay. Just sync then? [18:19:59] yeah [18:20:00] Niharika: mine is not very testable either, i tried a couple edits and it all looks good [18:20:02] Thanks :) [18:20:17] !log niharika29@tin Synchronized php-1.31.0-wmf.5/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 50s) [18:20:22] DMaza: Okay. [18:20:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:20:30] Amir1: Both of those are synced now. [18:21:01] Thanks [18:21:28] SMalyshev: Your patch is on mwdebug1002 now. [18:21:39] ok, checking [18:23:27] 10Operations, 10Traffic: LVS hosts should have static-mapped IPv6 on all virtual interfaces - https://phabricator.wikimedia.org/T179025#3710743 (10BBlack) [18:23:30] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Add AbuseFilterSlow channel to monolog T178853 (duration: 00m 50s) [18:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:23:38] T178853: Add new channel to monolog (AbuseFilterSlow) - https://phabricator.wikimedia.org/T178853 [18:23:49] Niharika: looks fine to me [18:24:24] 10Operations, 10Traffic: LVS IPv6 IPs should all be recorded in DNS - https://phabricator.wikimedia.org/T179026#3710760 (10BBlack) [18:24:32] 10Operations, 10Traffic: LVS hosts should have static-mapped IPv6 on all virtual interfaces - https://phabricator.wikimedia.org/T179025#3710774 (10BBlack) [18:24:34] 10Operations, 10Traffic: LVS IPv6 IPs should all be recorded in DNS - https://phabricator.wikimedia.org/T179026#3710773 (10BBlack) [18:25:52] !log niharika29@tin Synchronized wmf-config/Wikibase.php: Make using CirrusSearch engine default for wbsearchentities T175741 (duration: 00m 48s) [18:25:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:26:00] SMalyshev: Synced. [18:26:00] T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata - https://phabricator.wikimedia.org/T175741 [18:26:41] 10Operations, 10Traffic: Puppetize LVS interface IP sets per-DC for easy use in ferm rules - https://phabricator.wikimedia.org/T179027#3710778 (10BBlack) [18:26:56] Niharika: thanks! [18:27:37] RECOVERY - Router interfaces on cr1-codfw is OK: OK: host 208.80.153.192, interfaces up: 121, down: 0, dormant: 0, excluded: 0, unused: 0 [18:29:10] 10Operations, 10Traffic: Puppetize LVS interface IP sets per-DC for easy use in ferm rules - https://phabricator.wikimedia.org/T179027#3710800 (10Ottomata) [18:29:57] RECOVERY - Router interfaces on cr2-ulsfo is OK: OK: host 198.35.26.193, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [18:30:47] (03PS3) 10Ottomata: Add LVS for druid-public-overlord indexing service [puppet] - 10https://gerrit.wikimedia.org/r/386427 (https://phabricator.wikimedia.org/T176223) [18:33:09] !log niharika29@tin Synchronized php-1.31.0-wmf.4/extensions/Wikidata: Allow specifying a replacement usage type in disabledUsageAspects T178153 (duration: 02m 10s) [18:33:13] (03PS2) 10Niharika29: Make disabled usage aspect use the new config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386408 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [18:33:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:33:17] T178153: make disabledUsageAspects coarse graining - https://phabricator.wikimedia.org/T178153 [18:33:25] (03CR) 10Niharika29: [C: 032] "SWAT." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386408 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [18:34:12] (03PS2) 10Niharika29: Enable fine grained usage tracking on statement usage wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386421 (https://phabricator.wikimedia.org/T172914) (owner: 10Hoo man) [18:34:32] (03Merged) 10jenkins-bot: Make disabled usage aspect use the new config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386408 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [18:36:14] (03CR) 10jenkins-bot: Make disabled usage aspect use the new config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386408 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [18:36:30] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Make disabled usage aspect use the new config T172914 (duration: 00m 50s) [18:36:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:36:38] T172914: mw.wikibase.entity: Use __index to lazy register entity usages - https://phabricator.wikimedia.org/T172914 [18:37:20] (03PS3) 10Niharika29: Enable fine grained usage tracking on statement usage wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386421 (https://phabricator.wikimedia.org/T172914) (owner: 10Hoo man) [18:37:24] (03CR) 10Niharika29: [C: 032] Enable fine grained usage tracking on statement usage wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386421 (https://phabricator.wikimedia.org/T172914) (owner: 10Hoo man) [18:39:11] (03Merged) 10jenkins-bot: Enable fine grained usage tracking on statement usage wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386421 (https://phabricator.wikimedia.org/T172914) (owner: 10Hoo man) [18:39:19] (03CR) 10jenkins-bot: Enable fine grained usage tracking on statement usage wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386421 (https://phabricator.wikimedia.org/T172914) (owner: 10Hoo man) [18:44:44] !log reprepro: uploaded nginx-1.13.16-2+wmf1 packages to stretch and jessie [18:44:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:45:44] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Enable fine grained usage tracking on statement usage wikis T172914 (duration: 00m 50s) [18:45:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:45:51] T172914: mw.wikibase.entity: Use __index to lazy register entity usages - https://phabricator.wikimedia.org/T172914 [18:45:53] !log s/1\.13\.16/1.13.6/ [18:45:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:46:08] 10Operations, 10Puppet, 10User-Joe: Puppet: Use of 'import' has been discontinued in favor of a manifest directory. - https://phabricator.wikimedia.org/T179023#3710939 (10herron) To be clear this error doesn't occur when environment is set to "future". Still we might want to think about environment naming a... [18:46:32] Niharika: Thanks a lot [18:46:37] 10Operations, 10Puppet, 10User-Joe: Puppet: Use of 'import' has been discontinued in favor of a manifest directory. - https://phabricator.wikimedia.org/T179023#3710941 (10herron) p:05Normal>03Low [18:46:45] Amir1: You're welcome! :) [18:46:57] SWAT's all done. [18:47:28] 10Operations, 10Pybal, 10Traffic, 10netops, 10Patch-For-Review: Frequent RST returned by appservers to LVS hosts - https://phabricator.wikimedia.org/T163674#3710945 (10BBlack) patch above released with `nginx-1.13.6-2+wmf1`, so we're capable of experimentation now. Flag isn't turned on anywhere yet. [19:00:04] no_justification: My dear minions, it's time we take the moon! Just kidding. Time for MediaWiki train deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20171025T1900). [19:00:04] No GERRIT patches in the queue for this window AFAICS. [19:01:19] 10Operations, 10ops-eqiad, 10DBA: db1101 crashed - memory errors - https://phabricator.wikimedia.org/T178383#3711007 (10Marostegui) And now almost 50 alters running at the same time [19:05:49] Hi Jeff_Green [19:06:35] Jeff_Green: Do you mind if I take some of your time discussing your cluster usage? [19:11:09] (03PS1) 10Smalyshev: Add negative weight to disambig entities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386464 (https://phabricator.wikimedia.org/T148411) [19:12:37] (03CR) 10jerkins-bot: [V: 04-1] Add negative weight to disambig entities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386464 (https://phabricator.wikimedia.org/T148411) (owner: 10Smalyshev) [19:13:16] hey joal, sure [19:13:49] Jeff_Green: I have seen you regularly read big amount of webrequest data [19:14:11] yes, I'm restoring fundraising data lost by kafaktee [19:14:20] 10Operations, 10Puppet, 10User-Joe: Puppet: Error: Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `known_resource_types' - https://phabricator.wikimedia.org/T179033#3711031 (10herron) [19:14:42] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe, 10cloud-services-team (FY2017-18): Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3652273 (10herron) [19:14:43] (03PS2) 10Smalyshev: Add negative weight to disambig entities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386464 (https://phabricator.wikimedia.org/T148411) [19:15:33] (03PS1) 10BryanDavis: maintain-views: add additional log types [puppet] - 10https://gerrit.wikimedia.org/r/386465 (https://phabricator.wikimedia.org/T178752) [19:15:55] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe, 10cloud-services-team (FY2017-18): Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3652273 (10herron) [19:16:07] Jeff_Green: So the output string format is actually what you want I guess, and it wouldn't be interesting to store that as parquet [19:16:26] Jeff_Green: another thing: I think you are interested in webrequest coming to the text cache only [19:16:29] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe, 10cloud-services-team (FY2017-18): Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3711056 (10herron) [19:17:00] true, we're looking for tsv ultimately, that said I don't know what you mean by parquet [19:18:11] Jeff_Green: Your current request read both tet and upload, which double the size of data read (half a month of webrequest text + uplaod are about 20Tb) [19:18:37] ah you're right, somehow lost the text partition specification on my query, I have one more to do and will fix it for the next one [19:18:37] 10Operations, 10Puppet, 10User-Joe: Puppet: Error: Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `known_resource_types' - https://phabricator.wikimedia.org/T179033#3711060 (10herron) [19:18:53] Jeff_Green: parquet is an analytics-oriented file formqat, that would be very interesting if you were to use your data in hadoop (or other systems able to read it) [19:19:03] 10Operations, 10Puppet, 10User-Joe: Puppet: Error: Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `known_resource_types' - https://phabricator.wikimedia.org/T179033#3711031 (10herron) [19:19:36] joal: ok, that may be interesting when we get to redesigning the pipeline, I'm not sure [19:20:22] Jeff_Green: ok - I mention that because the jobs you lauch are very big - so making them half the size is not small optimization [19:20:47] (03CR) 10Chad: [C: 032] group1 to wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386392 (owner: 10Chad) [19:21:57] i think ultimately we need a separate topic, we only need a small subset of what's in text [19:22:01] (03Merged) 10jenkins-bot: group1 to wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386392 (owner: 10Chad) [19:22:10] (03CR) 10jenkins-bot: group1 to wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386392 (owner: 10Chad) [19:22:29] i suspect that's part of why kafkatee flaked out, don't know for sure though [19:22:51] also Jeff_Green, i would be great if you ran those huge jobs in the nice queue, since they are slow batches and don't need to finish very fast: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Queries#Run_long_queries_in_a_screen_session_and_in_the_nice_queue [19:23:53] ok. hopefully I have just one more to go. I thought I was already done, but there were some missing dates [19:24:34] (03PS6) 10BBlack: Global: Turn off ethernet flow for all interfaces at boot time [puppet] - 10https://gerrit.wikimedia.org/r/379799 [19:24:36] (03PS3) 10BBlack: Global: runtime disable ethernet flow on fresh install [puppet] - 10https://gerrit.wikimedia.org/r/381017 [19:24:38] (03PS7) 10BBlack: LVS: Disable LRO [puppet] - 10https://gerrit.wikimedia.org/r/379800 [19:24:40] (03PS9) 10BBlack: Caches: Disable LRO [puppet] - 10https://gerrit.wikimedia.org/r/379801 [19:26:16] PROBLEM - DPKG on restbase2005 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:27:03] (03CR) 10BBlack: [C: 032] Global: Turn off ethernet flow for all interfaces at boot time [puppet] - 10https://gerrit.wikimedia.org/r/379799 (owner: 10BBlack) [19:28:09] !log demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.5 [19:28:16] RECOVERY - DPKG on restbase2005 is OK: All packages OK [19:28:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:09] joal: is there a way to move a running query into the nice queue? [19:33:18] there is Jeff_Green :) [19:33:24] 10Operations, 10vm-requests: Request VM for webperf (metrics processing) - https://phabricator.wikimedia.org/T179036#3711107 (10Krinkle) [19:33:33] 10Operations, 10vm-requests, 10Performance-Team (Radar): Request VM for webperf (metrics processing) - https://phabricator.wikimedia.org/T179036#3711119 (10Krinkle) [19:33:41] Jeff_Green: on a stat machine: yarn application --movetoqueue APP_ID --queue nice [19:33:47] 10Operations, 10vm-requests, 10Performance-Team (Radar): Request VM for webperf (metrics processing) - https://phabricator.wikimedia.org/T179036#3711107 (10Krinkle) [19:34:02] 10Operations, 10Performance-Team, 10monitoring: Consolidate performance website and related software - https://phabricator.wikimedia.org/T158837#3711123 (10Krinkle) [19:34:05] 10Operations, 10vm-requests, 10Performance-Team (Radar): Request VM for webperf (metrics processing) - https://phabricator.wikimedia.org/T179036#3711107 (10Krinkle) [19:35:44] * Jeff_Green tries to figure out where to get APP_ID since it's past the screen backscroll buffer... [19:36:14] 10Operations, 10vm-requests, 10Performance-Team (Radar): Request VM for webperf (metrics processing) - https://phabricator.wikimedia.org/T179036#3711129 (10Krinkle) [19:36:14] Jeff_Green: I use yarn UI for that: yarn.wikimedia.org (LDAP login) [19:36:47] Jeff_Green: your currently running app_id: application_1504006918778_219385 [19:36:57] cool, looking at yarn too. thx! [19:37:57] done! [19:39:14] Thanks Jeff_Green :) [19:39:44] also Jeff_Green, please don't forget webrequest_source partition, it's actually very important for data size :) [19:40:50] (03PS10) 10Ottomata: Set up Kafka MirrorMaker from main -> jumbo in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/384586 (https://phabricator.wikimedia.org/T177216) [19:41:53] yep! [19:46:47] (03PS11) 10Ottomata: Set up Kafka MirrorMaker from main -> jumbo in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/384586 (https://phabricator.wikimedia.org/T177216) [19:49:44] (03PS12) 10Ottomata: Set up Kafka MirrorMaker from main -> jumbo in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/384586 (https://phabricator.wikimedia.org/T177216) [20:00:05] gwicke, cscott, arlolra, subbu, bearND, halfak, and Amir1: I, the Bot under the Fountain, allow thee, The Deployer, to do Services – Parsoid / OCG / Citoid / Mobileapps / ORES / … deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20171025T2000). [20:00:05] No GERRIT patches in the queue for this window AFAICS. [20:05:17] PROBLEM - pdfrender on scb1001 is CRITICAL: connect to address 10.64.0.16 and port 5252: Connection refused [20:08:09] No ORES deployment today. [20:21:09] !log demon@tin Synchronized php-1.31.0-wmf.5/extensions/ProofreadPage/: unbreak multiline text input (duration: 00m 53s) [20:21:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:34:56] !log restarted zuul to flush a long queue in experimental pipeline (no other changes enqueued) [20:35:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:37:04] (03CR) 10Andrew Bogott: [C: 032] git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [20:37:21] (03CR) 10Andrew Bogott: [C: 04-1] "This has a bug which I haven't yet diagnosed -- sometimes local changes are lost." [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [20:39:39] (03CR) 10Chad: "You're doing a force push and then reset -- I don't see how anything could survive" [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [20:49:39] (03PS13) 10Ottomata: Set up Kafka MirrorMaker from main -> jumbo in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/384586 (https://phabricator.wikimedia.org/T177216) [21:07:17] PROBLEM - Host cp4024 is DOWN: PING CRITICAL - Packet loss = 100% [21:12:46] PROBLEM - IPsec on cp2017 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:12:46] PROBLEM - IPsec on cp1064 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:12:47] PROBLEM - IPsec on kafka1012 is CRITICAL: Strongswan CRITICAL - ok: 112 connecting: cp4024_v4, cp4024_v6 [21:12:56] PROBLEM - IPsec on cp1071 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:12:57] PROBLEM - IPsec on cp2005 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:12:57] PROBLEM - IPsec on kafka1022 is CRITICAL: Strongswan CRITICAL - ok: 112 connecting: cp4024_v4, cp4024_v6 [21:12:57] PROBLEM - IPsec on cp1072 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:07] PROBLEM - IPsec on cp2022 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:16] PROBLEM - IPsec on cp1073 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:16] PROBLEM - IPsec on cp1063 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:16] PROBLEM - IPsec on cp2026 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:26] PROBLEM - IPsec on cp2002 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:26] PROBLEM - IPsec on kafka1018 is CRITICAL: Strongswan CRITICAL - ok: 112 connecting: cp4024_v4, cp4024_v6 [21:13:26] PROBLEM - IPsec on cp2011 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:26] PROBLEM - IPsec on cp2014 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:26] PROBLEM - IPsec on cp2020 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:27] PROBLEM - IPsec on cp2008 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:27] PROBLEM - IPsec on cp1050 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:28] PROBLEM - IPsec on cp2024 is CRITICAL: Strongswan CRITICAL - ok: 68 not-conn: cp4024_v4, cp4024_v6 [21:13:36] PROBLEM - IPsec on cp1049 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:37] PROBLEM - IPsec on cp1048 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:37] PROBLEM - IPsec on cp1062 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:37] PROBLEM - IPsec on kafka1013 is CRITICAL: Strongswan CRITICAL - ok: 112 connecting: cp4024_v4, cp4024_v6 [21:13:46] PROBLEM - IPsec on cp1099 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:13:46] PROBLEM - IPsec on kafka1020 is CRITICAL: Strongswan CRITICAL - ok: 112 connecting: cp4024_v4, cp4024_v6 [21:13:47] PROBLEM - IPsec on kafka1014 is CRITICAL: Strongswan CRITICAL - ok: 112 connecting: cp4024_v4, cp4024_v6 [21:13:56] PROBLEM - IPsec on cp1074 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 [21:14:18] 10Operations, 10Ops-Access-Requests, 10DBA, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3711314 (10madhuvishy) @jcrespo @bd808 I looked at the accounts set up we have now, and it looks like the labsdbadmin user is... [21:15:03] (03PS6) 10Ayounsi: eqsin revdns: strawman subnet plan [dns] - 10https://gerrit.wikimedia.org/r/385402 (https://phabricator.wikimedia.org/T156256) (owner: 10BBlack) [21:15:36] (03PS1) 10Bearloga: R, Shiny Server, and Discovery Computing fixes [puppet] - 10https://gerrit.wikimedia.org/r/386536 (https://phabricator.wikimedia.org/T178096) [21:15:38] (03CR) 10Ayounsi: [C: 032] eqsin revdns: strawman subnet plan [dns] - 10https://gerrit.wikimedia.org/r/385402 (https://phabricator.wikimedia.org/T156256) (owner: 10BBlack) [21:24:18] 10Operations, 10ops-ulsfo, 10Traffic: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3711338 (10BBlack) 05Resolved>03Open `cp4024` died randomly today. I've left it alone other than to connect to the console and verify no response there. `21:07 < icinga-wm> PROBLEM - Host cp4024 is DOW... [21:25:49] ACKNOWLEDGEMENT - IPsec on cp1048 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 Brandon Black T174891 [21:25:50] ACKNOWLEDGEMENT - IPsec on cp1049 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 Brandon Black T174891 [21:25:50] ACKNOWLEDGEMENT - IPsec on cp1050 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 Brandon Black T174891 [21:25:50] ACKNOWLEDGEMENT - IPsec on cp1062 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 Brandon Black T174891 [21:25:50] ACKNOWLEDGEMENT - IPsec on cp1063 is CRITICAL: Strongswan CRITICAL - ok: 54 not-conn: cp4024_v4, cp4024_v6 Brandon Black T174891 [21:26:19] way to go icinga-wm :P [21:26:48] restarting Cassandra, 2001-c [21:29:56] PROBLEM - cassandra-c CQL 10.192.16.164:9042 on restbase2001 is CRITICAL: connect to address 10.192.16.164 and port 9042: Connection refused [21:30:56] RECOVERY - cassandra-c CQL 10.192.16.164:9042 on restbase2001 is OK: TCP OK - 0.036 second response time on 10.192.16.164 port 9042 [21:33:40] !log experimental - disabled LRO on lvs4001+lvs4002 [21:33:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:35:13] (03PS1) 10Ayounsi: Reserve IPs for eqsin RIPE atlas [dns] - 10https://gerrit.wikimedia.org/r/386538 [21:40:33] 10Operations, 10ops-eqiad, 10netops: Setup eqsin atlas anchor - https://phabricator.wikimedia.org/T179042#3711364 (10ayounsi) [21:40:36] (03CR) 10Ayounsi: [C: 032] Reserve IPs for eqsin RIPE atlas [dns] - 10https://gerrit.wikimedia.org/r/386538 (owner: 10Ayounsi) [21:48:19] !log mobrovac@tin Started deploy [restbase/deploy@60ce036]: Revert double-processing of all summaries to all except WPs [21:48:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:48:31] (03PS1) 10Rush: openstack: add new puppetmaster profile dummy secrets [labs/private] - 10https://gerrit.wikimedia.org/r/386540 [21:49:51] (03PS2) 10Rush: openstack: add new puppetmaster profile dummy secrets [labs/private] - 10https://gerrit.wikimedia.org/r/386540 [21:50:58] PROBLEM - Restbase root url on restbase1007 is CRITICAL: connect to address 10.64.0.223 and port 7231: Connection refused [21:51:31] (03CR) 10Rush: [V: 032 C: 032] openstack: add new puppetmaster profile dummy secrets [labs/private] - 10https://gerrit.wikimedia.org/r/386540 (owner: 10Rush) [21:52:26] PROBLEM - Check systemd state on restbase1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [21:52:41] !log mobrovac@tin Started deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2 [21:52:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:53:22] 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Requesting access to ops for aborrero - https://phabricator.wikimedia.org/T178809#3711396 (10chasemp) >>! In T178809#3703943, @Dzahn wrote: > We can start with getting him on non-public IRC channels and the desired mailing lists. Also wikitech LDAP us... [21:53:26] RECOVERY - Check systemd state on restbase1007 is OK: OK - running: The system is fully operational [21:53:36] 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Requesting access to ops for aborrero - https://phabricator.wikimedia.org/T178809#3711398 (10chasemp) 05Open>03Resolved a:03chasemp [21:53:57] RECOVERY - Restbase root url on restbase1007 is OK: HTTP OK: HTTP/1.1 200 - 15742 bytes in 0.005 second response time [22:01:50] !log mobrovac@tin Finished deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2 (duration: 09m 09s) [22:01:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:38:20] (03CR) 10Andrew Bogott: [C: 04-1] "> You're doing a force push and then reset -- I don't see how anything could survive" [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [22:42:13] !log running migratePreferences.php on group1 wikis [22:42:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:45:42] (03PS8) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [22:51:55] (03CR) 10BryanDavis: "The original bash would halt if updating /var/lib/git/operations/puppet failed for some reason. I'm not 100% sure how, but there might be " [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) (owner: 10Andrew Bogott) [22:54:08] (03PS1) 10Dmaza: Change threshold for slow AbuseFilter logging to 800ms [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386547 (https://phabricator.wikimedia.org/T179039) [23:00:00] (03PS7) 10Andrew Bogott: git-sync-upstream: rewrite in python [puppet] - 10https://gerrit.wikimedia.org/r/386318 (https://phabricator.wikimedia.org/T177944) [23:00:02] (03PS9) 10Andrew Bogott: git-sync-upstream: perform rebase in a separate, temporary workdir [puppet] - 10https://gerrit.wikimedia.org/r/386331 (https://phabricator.wikimedia.org/T177944) [23:00:04] addshore, hashar, anomie, RainbowSprinkles, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: (Dis)respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20171025T2300). Please do the needful. [23:00:04] Deskana and Smalyshev: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [23:00:13] here [23:00:22] Me too! [23:04:10] now we only need some SWATter :) [23:04:38] mm.. I wonder why I was not notified, I have a patch schedule for this swat [23:05:11] DMaza: I think if you add it too close to the time, the bot doesn't get it [23:05:20] ah!.. that was it [23:05:24] jouncebot: reload [23:05:49] jouncebot: refresh [23:05:52] I refreshed my knowledge about deployments. [23:12:13] Any human available for the Evening SWAT? [23:15:37] 10Operations, 10ops-ulsfo, 10Traffic: decom cp40(09|1[078]) - https://phabricator.wikimedia.org/T178815#3711530 (10RobH) a:05BBlack>03RobH [23:18:11] so, looks like it's not happening today? [23:20:18] (03CR) 10Ayounsi: [C: 032] admin: Add legoktm's new ed25519 key [puppet] - 10https://gerrit.wikimedia.org/r/384634 (owner: 10Legoktm) [23:20:23] (03PS2) 10Ayounsi: admin: Add legoktm's new ed25519 key [puppet] - 10https://gerrit.wikimedia.org/r/384634 (owner: 10Legoktm) [23:21:49] SMalyshev: I guess not, Do you know off anyone that is usually available at this time? [23:22:40] I can do it in a few minutes [23:22:52] yay! thanks legoktm [23:23:06] legoktm: cool! [23:27:30] might be a few more minutes, I'm trying to use my new ssh key [23:33:05] (03CR) 10Legoktm: [C: 032] Add negative weight to disambig entities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386464 (https://phabricator.wikimedia.org/T148411) (owner: 10Smalyshev) [23:33:09] (03CR) 10Legoktm: [C: 032] Change threshold for slow AbuseFilter logging to 800ms [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386547 (https://phabricator.wikimedia.org/T179039) (owner: 10Dmaza) [23:33:15] Deskana: still around? [23:33:26] legoktm: Yep. [23:34:15] (03Merged) 10jenkins-bot: Add negative weight to disambig entities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386464 (https://phabricator.wikimedia.org/T148411) (owner: 10Smalyshev) [23:34:17] (03Merged) 10jenkins-bot: Change threshold for slow AbuseFilter logging to 800ms [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386547 (https://phabricator.wikimedia.org/T179039) (owner: 10Dmaza) [23:35:05] DMaza: SMalyshev: your changes are live on mwdebug1002 [23:35:27] ok [23:36:19] (03CR) 10jenkins-bot: Add negative weight to disambig entities [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386464 (https://phabricator.wikimedia.org/T148411) (owner: 10Smalyshev) [23:37:18] welp, mine didn't break anything.. I can't really test it other than making sure edits are still working [23:37:37] and they are.. so.. it looks good :) [23:38:14] that's fine, if things start breaking I'll be around for another hour or two [23:38:20] * robh old yeller's cp4009 [23:38:24] woooo [23:39:24] it should be fine, it is nothing that could break [23:39:27] !log legoktm@tin Synchronized wmf-config/abusefilter.php: Change threshold for slow AbuseFilter logging to 800ms - T179039 (duration: 00m 50s) [23:39:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:39:36] T179039: Change threshold for slow AbuseFilter logging from 500 ms to 800 ms - https://phabricator.wikimedia.org/T179039 [23:40:11] legoktm: hmm doesn't really work... looks like the code that was needed didn't get into this branch :( [23:40:21] SMalyshev: ok, should I revert it? [23:40:22] so revert for now and I'll submit it later when next branch is cut [23:40:37] I thought it was in but apparently I was wrong, so have to wait [23:41:01] legoktm: revert for now, I'll resubmit it later [23:41:11] (03PS1) 10Legoktm: Revert "Add negative weight to disambig entities" legoktm: hmm doesn't really work... looks like the code that was needed didn't get into this branch :( so revert for now and I'll submit it later when next branch is cut [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386552 [23:41:21] (03PS2) 10Legoktm: Revert "Add negative weight to disambig entities" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386552 [23:41:25] (03CR) 10Legoktm: [C: 032] Revert "Add negative weight to disambig entities" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386552 (owner: 10Legoktm) [23:42:41] (03Merged) 10jenkins-bot: Revert "Add negative weight to disambig entities" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386552 (owner: 10Legoktm) [23:43:56] Deskana: should be live on mwdebug1002 now [23:43:58] (03PS1) 10MaxSem: Enable Unicode section links on Russian projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386553 (https://phabricator.wikimedia.org/T175725) [23:44:06] Checking. [23:45:26] legoktm: Yep, it's working. [23:45:38] awesome [23:45:56] legoktm: did you deploy the revert on mwdebug? [23:46:05] SMalyshev: I didn't [23:46:23] SMalyshev: it's reverted on mwdebug now [23:46:52] yep, confirming, everything is fine now [23:47:00] legoktm: thanks, sorry for broken patch [23:47:28] no worries [23:48:16] !log legoktm@tin Synchronized php-1.31.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: MWInternalLinkContextItem: increase specificity to override OOUI changes - T178933 (duration: 00m 50s) [23:48:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:25] T178933: Images shown in the link tool are very short and wide at cswiki - https://phabricator.wikimedia.org/T178933 [23:48:25] Deskana: ^ [23:49:02] legoktm: Excellent. Thank you! [23:49:04] np [23:49:13] (03PS1) 10Smalyshev: Revert "Revert "Add negative weight to disambig entities"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/386554 (https://phabricator.wikimedia.org/T148411)