[00:17:47] PROBLEM - MariaDB Slave Lag: s3 on db2098 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1063.95 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave [01:47:09] RECOVERY - MariaDB Slave Lag: s3 on db2098 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave [01:53:17] PROBLEM - BFD status on cr1-eqiad is CRITICAL: CRIT: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [06:59:07] (03PS1) 10Legoktm: Improve Python 3 compatibility [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561262 [07:11:44] !log Restart wikibugs - T241109 [07:11:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:11:53] T241109: wikibugs needs restart almost everyday - https://phabricator.wikimedia.org/T241109 [07:14:20] 10Operations, 10Wikibugs: wikibugs needs restart almost everyday - https://phabricator.wikimedia.org/T241109 (10Marostegui) I have had to restart `wikibugs` today. The log only showed: ` 019-12-31 07:11:02,879 - urllib3.connectionpool - DEBUG - https://phabricator.wikimedia.org:443 "POST /api/feed.query HTTP/1... [07:40:37] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10Marostegui) Thanks for catching that Jaime, I have created this: https://wikitech.wikimedia.org/w/index.php?title=MariaDB&type=revision&diff=1849295&oldid=1847866 [07:46:12] !log Upgrade db2126 db2125 db2108 db2107 [07:46:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:49:08] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [07:58:20] (03PS1) 10Marostegui: control-mariadb*: Upgrade version [software] - 10https://gerrit.wikimedia.org/r/561264 (https://phabricator.wikimedia.org/T193224) [08:06:18] 10Operations, 10Wikibugs: wikibugs needs restart almost everyday - https://phabricator.wikimedia.org/T241109 (10ema) FTR, on 2019-12-29 I also restarted wikibugs due to phab comments not showing up on irc. I did not check the logs at the time. [08:06:30] 10Operations, 10Wikibugs: wikibugs needs restart almost everyday - https://phabricator.wikimedia.org/T241109 (10ema) p:05Triage→03Normal [08:11:23] 10Operations, 10Wikibugs: wikibugs needs restart almost everyday - https://phabricator.wikimedia.org/T241109 (10Marostegui) >>! In T241109#5758345, @valhallasw wrote: > > Not entirely sure how to fix this yet, though. We can probably handle that error and just resynchronize as we would do on start. That does... [08:24:41] !log Upgrade db1107 from 10.4.10 to 10.4.11 [08:24:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:25:03] !log Upgrade db2128 [08:25:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:26:16] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:33:52] (03Abandoned) 10Hashar: gerrit: enable AccountDeactivator [puppet] - 10https://gerrit.wikimedia.org/r/505218 (https://phabricator.wikimedia.org/T218654) (owner: 10Hashar) [08:37:26] (03Abandoned) 10Hashar: Output summary of build errors upon completion [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/520751 (owner: 10Hashar) [08:38:26] (03Abandoned) 10Hashar: Add image metadata to Jinja context [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/520772 (owner: 10Hashar) [08:38:59] (03Abandoned) 10Hashar: profile: point to real modules for specs [puppet] - 10https://gerrit.wikimedia.org/r/480957 (owner: 10Hashar) [08:39:23] (03CR) 10Hashar: "I could not find any appropriate time to think about it :-\" [puppet] - 10https://gerrit.wikimedia.org/r/480957 (owner: 10Hashar) [08:45:01] (03Abandoned) 10Hashar: contint: ensure docker-ce is available [puppet] - 10https://gerrit.wikimedia.org/r/538938 (owner: 10Hashar) [08:58:04] (03CR) 10Hashar: "Feel free to amend the distribution to 'buster-wikimedia' :)" [debs/poolcounter-prometheus-exporter] - 10https://gerrit.wikimedia.org/r/553736 (owner: 10Hashar) [08:59:46] (03CR) 10Jcrespo: [C: 03+2] contint: downgrade docker on Stretch to match Jessie [puppet] - 10https://gerrit.wikimedia.org/r/547313 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [09:00:01] (03PS2) 10Jcrespo: contint: downgrade docker on Stretch to match Jessie [puppet] - 10https://gerrit.wikimedia.org/r/547313 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [09:03:51] (03PS1) 10Ema: mtail: port varnishbackendtiming to ATS [puppet] - 10https://gerrit.wikimedia.org/r/561266 (https://phabricator.wikimedia.org/T233474) [09:05:45] (03CR) 10jerkins-bot: [V: 04-1] mtail: port varnishbackendtiming to ATS [puppet] - 10https://gerrit.wikimedia.org/r/561266 (https://phabricator.wikimedia.org/T233474) (owner: 10Ema) [09:12:13] (03PS2) 10Ema: mtail: port varnishbackendtiming to ATS [puppet] - 10https://gerrit.wikimedia.org/r/561266 (https://phabricator.wikimedia.org/T233474) [09:15:37] (03CR) 10Ema: "pcc looks sane: https://puppet-compiler.wmflabs.org/compiler1003/20131/" [puppet] - 10https://gerrit.wikimedia.org/r/561266 (https://phabricator.wikimedia.org/T233474) (owner: 10Ema) [10:04:02] 10Operations, 10DBA, 10MediaWiki-General, 10Patch-For-Review: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10Marostegui) I have upgraded db1107 from 10.4.10 to the latest version: 10.4.11 [10:04:05] (03CR) 10Marostegui: [C: 03+2] control-mariadb*: Upgrade version [software] - 10https://gerrit.wikimedia.org/r/561264 (https://phabricator.wikimedia.org/T193224) (owner: 10Marostegui) [10:04:35] (03Merged) 10jenkins-bot: control-mariadb*: Upgrade version [software] - 10https://gerrit.wikimedia.org/r/561264 (https://phabricator.wikimedia.org/T193224) (owner: 10Marostegui) [10:08:31] PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 43153016 and 3 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:10:07] PROBLEM - Postgres Replication Lag on maps1001 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 19242104 and 2 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:10:19] RECOVERY - Postgres Replication Lag on maps2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 0 and 46 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:11:55] RECOVERY - Postgres Replication Lag on maps1001 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 24632 and 68 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:16:27] (03PS1) 10Elukey: graphite::wmcs::archiver: fix daily cronspam [puppet] - 10https://gerrit.wikimedia.org/r/561273 [10:22:53] RECOVERY - BFD status on cr1-eqiad is OK: OK: UP: 11 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:23:33] !log execute 'clear bfd session address fe80::5e5e:ab00:d3d:85ce' on cr3-knams - T240659 [10:23:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:23:41] T240659: BFD session alerts due to inconsistent status on cr3-knams - https://phabricator.wikimedia.org/T240659 [10:34:39] !log Upgrade db2111 and db2123 [10:34:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:36:48] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [10:49:07] !log Upgrade db1108 with elukey [10:49:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:53:12] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [11:27:52] (03CR) 10Andrew Bogott: [C: 03+2] graphite::wmcs::archiver: fix daily cronspam [puppet] - 10https://gerrit.wikimedia.org/r/561273 (owner: 10Elukey) [11:35:16] !log Upgrade db2073 [11:35:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:42:57] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [11:48:29] !log Upgrade db2119 and db2106 [11:48:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:53:28] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [12:02:46] (03PS1) 10Andrew Bogott: Horizon is now running OpenStack 'train' [puppet] - 10https://gerrit.wikimedia.org/r/561275 [12:02:48] (03PS1) 10Andrew Bogott: Move cloudcontrol2003-dev to openstack 'pike' [puppet] - 10https://gerrit.wikimedia.org/r/561276 (https://phabricator.wikimedia.org/T241347) [12:02:50] (03PS1) 10Andrew Bogott: move codfw1-dev cluster to openstack 'pike' [puppet] - 10https://gerrit.wikimedia.org/r/561277 (https://phabricator.wikimedia.org/T241347) [12:06:39] (03CR) 10Andrew Bogott: [C: 03+2] Horizon is now running OpenStack 'train' [puppet] - 10https://gerrit.wikimedia.org/r/561275 (owner: 10Andrew Bogott) [12:11:28] (03PS4) 10Andrew Bogott: keystone/pike: remove obsolete filter from paste.ini [puppet] - 10https://gerrit.wikimedia.org/r/560375 (https://phabricator.wikimedia.org/T241347) [12:12:08] (03PS4) 10Andrew Bogott: nova/pike: update policy.json for new Pike policy changes [puppet] - 10https://gerrit.wikimedia.org/r/560376 (https://phabricator.wikimedia.org/T241347) [12:12:33] (03CR) 10Andrew Bogott: [C: 03+2] keystone/pike: remove obsolete filter from paste.ini [puppet] - 10https://gerrit.wikimedia.org/r/560375 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:13:09] (03CR) 10Andrew Bogott: [C: 03+2] nova/pike: update policy.json for new Pike policy changes [puppet] - 10https://gerrit.wikimedia.org/r/560376 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:16:50] (03PS1) 10Andrew Bogott: nova: Remove manifest for the spiceproxy service [puppet] - 10https://gerrit.wikimedia.org/r/561278 [12:23:18] (03CR) 10Andrew Bogott: [C: 03+2] nova: Remove manifest for the spiceproxy service [puppet] - 10https://gerrit.wikimedia.org/r/561278 (owner: 10Andrew Bogott) [12:32:06] (03PS1) 10Andrew Bogott: nova: add service manifests for pike [puppet] - 10https://gerrit.wikimedia.org/r/561279 (https://phabricator.wikimedia.org/T241347) [12:32:09] (03PS1) 10Andrew Bogott: glance: add service manifest for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561280 [12:32:11] (03PS1) 10Andrew Bogott: keystone: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561281 [12:32:13] (03PS1) 10Andrew Bogott: neutron: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561282 [12:32:15] (03PS1) 10Andrew Bogott: openstack clientpackages: add manifests for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561283 [12:33:06] (03CR) 10jerkins-bot: [V: 04-1] glance: add service manifest for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561280 (owner: 10Andrew Bogott) [12:33:27] (03CR) 10jerkins-bot: [V: 04-1] keystone: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561281 (owner: 10Andrew Bogott) [12:34:01] (03CR) 10jerkins-bot: [V: 04-1] neutron: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561282 (owner: 10Andrew Bogott) [12:37:22] (03PS2) 10Andrew Bogott: nova: add service manifests for pike [puppet] - 10https://gerrit.wikimedia.org/r/561279 (https://phabricator.wikimedia.org/T241347) [12:37:24] (03PS2) 10Andrew Bogott: glance: add service manifest for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561280 (https://phabricator.wikimedia.org/T241347) [12:37:26] (03PS2) 10Andrew Bogott: keystone: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561281 (https://phabricator.wikimedia.org/T241347) [12:37:28] (03PS2) 10Andrew Bogott: neutron: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561282 (https://phabricator.wikimedia.org/T241347) [12:37:30] (03PS2) 10Andrew Bogott: openstack clientpackages: add manifests for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561283 (https://phabricator.wikimedia.org/T241347) [12:38:44] (03CR) 10jerkins-bot: [V: 04-1] keystone: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561281 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:39:09] (03CR) 10Andrew Bogott: [C: 03+2] nova: add service manifests for pike [puppet] - 10https://gerrit.wikimedia.org/r/561279 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:39:18] (03CR) 10jerkins-bot: [V: 04-1] neutron: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561282 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:39:40] (03CR) 10Andrew Bogott: [C: 03+2] glance: add service manifest for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561280 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:41:03] (03CR) 10Andrew Bogott: [C: 03+2] "overriding jenkins because the only complaint is about importing network::constants" [puppet] - 10https://gerrit.wikimedia.org/r/561281 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:41:08] (03CR) 10Andrew Bogott: [V: 03+2 C: 03+2] keystone: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561281 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:47:55] (03PS3) 10Andrew Bogott: neutron: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561282 (https://phabricator.wikimedia.org/T241347) [12:48:46] (03CR) 10Andrew Bogott: [C: 03+2] neutron: add service manifests for openstack Pike [puppet] - 10https://gerrit.wikimedia.org/r/561282 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:48:58] (03CR) 10Andrew Bogott: [C: 03+2] openstack clientpackages: add manifests for openstack pike [puppet] - 10https://gerrit.wikimedia.org/r/561283 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:49:39] (03PS2) 10Andrew Bogott: Move cloudcontrol2003-dev to openstack 'pike' [puppet] - 10https://gerrit.wikimedia.org/r/561276 (https://phabricator.wikimedia.org/T241347) [12:50:33] (03CR) 10Andrew Bogott: [C: 03+2] Move cloudcontrol2003-dev to openstack 'pike' [puppet] - 10https://gerrit.wikimedia.org/r/561276 (https://phabricator.wikimedia.org/T241347) (owner: 10Andrew Bogott) [12:57:36] 10Operations, 10Puppet, 10DBA, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [13:09:32] 10Operations, 10DBA: Upgrade BIOS and firmware on db2076 - https://phabricator.wikimedia.org/T241647 (10Marostegui) [13:10:08] 10Operations, 10ops-codfw, 10DBA: Upgrade BIOS and firmware on db2076 - https://phabricator.wikimedia.org/T241647 (10Marostegui) p:05Triage→03Normal a:03Papaul [13:41:46] (03PS1) 10Majavah: Enable wgCiteResponsiveReferences on cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561286 (https://phabricator.wikimedia.org/T241304) [13:49:27] Hi, can someone please do T241648? Thanks [13:49:28] T241648: Special:BrokenRedirects shows displays an incorrect state on srwiki - https://phabricator.wikimedia.org/T241648 [13:49:41] Too I want wish happy holidays to all! [14:06:54] Zoranzoki21: I guess that won't be picked up today [14:17:35] (03PS1) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add additional packages [puppet] - 10https://gerrit.wikimedia.org/r/561289 (https://phabricator.wikimedia.org/T241347) [14:19:06] (03PS2) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add additional packages [puppet] - 10https://gerrit.wikimedia.org/r/561289 (https://phabricator.wikimedia.org/T241347) [15:23:01] PROBLEM - recommendation_api endpoints health on scb1002 is CRITICAL: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - bad seed) is CRITICAL: Test article.creation.translation - bad seed returned the unexpected status 500 (expecting: 404) https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [15:24:51] RECOVERY - recommendation_api endpoints health on scb1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [15:30:25] PROBLEM - recommendation_api endpoints health on scb2006 is CRITICAL: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - bad seed) is CRITICAL: Test article.creation.translation - bad seed returned the unexpected status 500 (expecting: 404) https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [15:32:15] PROBLEM - recommendation_api endpoints health on scb2001 is CRITICAL: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - bad seed) is CRITICAL: Test article.creation.translation - bad seed returned the unexpected status 500 (expecting: 404) https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [15:32:15] RECOVERY - recommendation_api endpoints health on scb2006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [15:34:05] RECOVERY - recommendation_api endpoints health on scb2001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [15:34:24] 10Operations, 10Wikibugs, 10Patch-For-Review: wikibugs needs restart almost everyday - https://phabricator.wikimedia.org/T241109 (10valhallasw) I have deployed this patchset; it needs some work (formatting etc) and the change should really go upstream to Fab; I'll discuss that with @Legoktm. For now I just w... [16:52:12] (03PS3) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add additional packages [puppet] - 10https://gerrit.wikimedia.org/r/561289 (https://phabricator.wikimedia.org/T241347) [16:54:19] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] aptrepo: thirdparty/openstack-pike-stretch: add additional packages [puppet] - 10https://gerrit.wikimedia.org/r/561289 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [16:57:10] !log import more openstack packages into reprepro (T241347) thirdparty/openstack-pike-stretch [16:57:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:57:20] T241347: upgrade cloud-vps openstack to Openstack version 'Pike' - https://phabricator.wikimedia.org/T241347 [17:04:28] (03PS1) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add additional packages again Additional packages are required in order for us to be able to install openstack glance, nova, keystone and neutron in cloudcontrol boxes. [puppet] - 10https://gerrit.wikimedia.org/r/561297 (https://phabricator.wikimedia.org/T241347) [17:05:41] (03CR) 10jerkins-bot: [V: 04-1] aptrepo: thirdparty/openstack-pike-stretch: add additional packages again Additional packages are required in order for us to be able to install openstack glance, nova, keystone and neutron in cloudcontrol boxes. [puppet] - 10https://gerrit.wikimedia.org/r/561297 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [17:05:58] (03PS2) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add additional packages again [puppet] - 10https://gerrit.wikimedia.org/r/561297 (https://phabricator.wikimedia.org/T241347) [17:10:36] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] aptrepo: thirdparty/openstack-pike-stretch: add additional packages again [puppet] - 10https://gerrit.wikimedia.org/r/561297 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [17:15:30] 10Operations, 10Discovery-Search, 10Elasticsearch, 10Traffic, 10Performance Issue: Sustained periods (2-4h) of bad latency on production-search eqiad - https://phabricator.wikimedia.org/T241421 (10dcausse) I believe this is caused by a bot sending a large amount of requests of type: `/w/api.php?format=js... [17:17:03] (03PS1) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add chardet and python-urllib3 [puppet] - 10https://gerrit.wikimedia.org/r/561298 (https://phabricator.wikimedia.org/T241347) [17:17:52] (03CR) 10jerkins-bot: [V: 04-1] aptrepo: thirdparty/openstack-pike-stretch: add chardet and python-urllib3 [puppet] - 10https://gerrit.wikimedia.org/r/561298 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [17:21:51] (03PS2) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add chardet and python-urllib3 [puppet] - 10https://gerrit.wikimedia.org/r/561298 (https://phabricator.wikimedia.org/T241347) [17:24:18] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] aptrepo: thirdparty/openstack-pike-stretch: add chardet and python-urllib3 [puppet] - 10https://gerrit.wikimedia.org/r/561298 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [17:29:30] (03PS1) 10Arturo Borrero Gonzalez: aptrepo: thirdparty/openstack-pike-stretch: add some more python libs [puppet] - 10https://gerrit.wikimedia.org/r/561299 (https://phabricator.wikimedia.org/T241347) [17:38:18] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] aptrepo: thirdparty/openstack-pike-stretch: add some more python libs [puppet] - 10https://gerrit.wikimedia.org/r/561299 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [17:43:06] (03Abandoned) 10Arturo Borrero Gonzalez: openstack: serverpackages: pike: stretch: pin some packages from stretch-bpo [puppet] - 10https://gerrit.wikimedia.org/r/560847 (https://phabricator.wikimedia.org/T241347) (owner: 10Arturo Borrero Gonzalez) [18:56:31] (03PS1) 10CDanis: block one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) [19:00:59] (03PS2) 10CDanis: block one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) [19:15:56] (03PS3) 10CDanis: block one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) [19:24:08] 10Operations, 10Traffic: two failing upload VTC tests - https://phabricator.wikimedia.org/T241653 (10CDanis) [19:24:38] (03PS4) 10CDanis: block one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) [19:35:23] (03PS5) 10CDanis: block one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) [19:41:10] (03PS6) 10CDanis: block search API traffic from one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) [19:42:06] (03CR) 10CDanis: [C: 03+2] block search API traffic from one U-A running on AWS [puppet] - 10https://gerrit.wikimedia.org/r/561300 (https://phabricator.wikimedia.org/T241421) (owner: 10CDanis) [19:43:51] (03PS1) 10CDanis: anchor regex to start [puppet] - 10https://gerrit.wikimedia.org/r/561304 (https://phabricator.wikimedia.org/T241421) [19:46:41] (03PS2) 10CDanis: AWS search block: anchor regex to start [puppet] - 10https://gerrit.wikimedia.org/r/561304 (https://phabricator.wikimedia.org/T241421) [19:47:26] (03CR) 10CDanis: [C: 03+2] AWS search block: anchor regex to start [puppet] - 10https://gerrit.wikimedia.org/r/561304 (https://phabricator.wikimedia.org/T241421) (owner: 10CDanis) [20:33:48] 10Operations, 10Traffic, 10Wikimedia-Apache-configuration, 10HTTPS: sec-warning page is Wikipedia-specific and dubiously worded - https://phabricator.wikimedia.org/T241656 (10Nemo_bis) [21:09:03] 10Operations, 10Traffic, 10Wikimedia-Apache-configuration, 10HTTPS: sec-warning page is Wikipedia-specific and dubiously worded - https://phabricator.wikimedia.org/T241656 (10DannyS712) The message is specified in https://gerrit.wikimedia.org/g/operations/puppet/+/5b1adde31e76da719f7a12211f9df6f20bd26b25/m... [21:13:31] 10Operations, 10Traffic, 10HTTPS: sec-warning page is Wikipedia-specific and dubiously worded - https://phabricator.wikimedia.org/T241656 (10Aklapper) [21:15:51] 10Operations, 10Traffic, 10HTTPS, 10Voice & Tone: sec-warning page is Wikipedia-specific and dubiously worded - https://phabricator.wikimedia.org/T241656 (10Aklapper) [23:01:17] HAPPY NEW YEAR TO EVERYONE! [23:02:57] Not yet! [23:05:59] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 53.36 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [23:17:24] In Serbia it is [23:20:17] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 71.39 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [23:57:29] Oh it is GRP_ [23:57:36] Oops wrong channel