[08:09:08] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1301.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201912020808_jiji_185958.log`. [08:15:20] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1287.eqiad.wmnet', 'mw1288.eqiad.wmnet', 'mw1289.eqiad.wmnet'] ` The log can be found in `/var/log/... [09:15:47] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1301.eqiad.wmnet'] ` and were **ALL** successful. [09:24:24] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1300.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201912020924_jiji_202065.log`. [10:19:32] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1275.eqiad.wmnet', 'mw1290.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [10:29:37] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1300.eqiad.wmnet'] ` and were **ALL** successful. [10:43:02] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1299.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201912021042_jiji_219639.log`. [11:46:47] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1287.eqiad.wmnet', 'mw1288.eqiad.wmnet', 'mw1289.eqiad.wmnet'] ` and were **ALL** successful. [11:49:16] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1299.eqiad.wmnet'] ` and were **ALL** successful. [11:51:30] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10fgiunchedi) [12:38:42] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1296.eqiad.wmnet', 'mw1298.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [12:46:57] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1275.eqiad.wmnet', 'mw1290.eqiad.wmnet'] ` and were **ALL** successful. [12:55:12] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [14:12:31] 10serviceops, 10Core Platform Team: MediaWiki::restInPeace: transaction round 'MediaWiki\Linter\RecordLintJob::run' still running - https://phabricator.wikimedia.org/T239607 (10jijiki) [14:51:38] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1296.eqiad.wmnet', 'mw1298.eqiad.wmnet'] ` and were **ALL** successful. [15:13:17] 10serviceops, 10Operations, 10Parsoid-PHP, 10Wikimedia-production-error: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10ssastry) >>! In T236833#5692042, @jcrespo wrote: > This is ongoing, so adding production error tag: This will be more or less ongoing forever. At best, we... [15:14:07] 10serviceops, 10Operations, 10Parsoid-PHP, 10Wikimedia-production-error: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10ssastry) >>! In T236833#5693660, @Joe wrote: > Patches are merged and the memory limit is now at 760 MB, as confirmed by the current OOMs. Not sure how mu... [15:36:04] 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): MediaWiki::restInPeace: transaction round 'MediaWiki\Linter\RecordLintJob::run' still running - https://phabricator.wikimedia.org/T239607 (10mobrovac) 05Open→03Resolved p:05Triage→03High a:03mobrovac [15:39:24] 10serviceops, 10Operations, 10Parsoid-PHP, 10Wikimedia-production-error: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10jcrespo) > if the expectation is that the production error tag will give this higher priority compared other Parsoid bugs, that is not going to be the case... [16:19:57] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1295.eqiad.wmnet', 'mw1294.eqiad.wmnet', 'mw1293.eqiad.wmnet'] ` The log can be found in `/var/log/... [17:43:35] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2240.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021743_dzahn_48992_mw224... [17:45:25] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2239.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021745_dzahn_49266_mw223... [17:57:39] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2238.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021757_dzahn_50879_mw223... [17:59:26] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2237.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021759_dzahn_51262_mw223... [18:54:51] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2240.codfw.wmnet'] ` and were **ALL** successful. [18:55:50] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2239.codfw.wmnet'] ` and were **ALL** successful. [19:01:51] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog, and 2 others: Resolve service instability due to excessive event loop blockage since starting PCS response pregeneration - https://phabricator.wikimedia.org/T229286 (10JMinor) [19:02:15] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2245.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021901_dzahn_65268_mw224... [19:03:03] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2234.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021902_dzahn_65614_mw223... [19:09:31] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2238.codfw.wmnet'] ` and were **ALL** successful. [19:11:00] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2237.codfw.wmnet'] ` and were **ALL** successful. [19:21:17] o/ i just tried to deploy staging eventgate-logging-extenernal with new tls proxy certs [19:21:31] but i think the envoy proxy container keeps dying [19:21:41] i'm not sure why, and i haven't been able to find any helpful logs yet either [19:21:51] how can I find out why a container is dying? [19:22:31] as far as I can tell the main app has come up fine [19:22:37] but i can't reach it on the non https port either [19:25:10] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2246.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021923_dzahn_71993_mw224... [19:25:37] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` mw2233.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021925_ariel_72277_mw223... [19:25:46] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` mw2232.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021925_ariel_72297_mw223... [19:26:03] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2247.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021925_dzahn_72347_mw224... [19:26:06] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` mw2231.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021925_ariel_72344_mw223... [19:26:18] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` mw2234.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021926_ariel_72413_mw223... [19:26:22] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2234.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2234.codfw.wmnet'] ` [19:27:17] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` mw2234.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021927_ariel_72531_mw223... [19:28:35] 10serviceops, 10Core Platform Team, 10Performance-Team, 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [19:30:07] 10serviceops, 10Core Platform Team, 10Performance-Team, 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [19:30:29] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2234.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2234.codfw.wmnet'] ` [19:31:55] 10serviceops, 10Phabricator, 10Documentation, 10Release-Engineering-Team (Development services), and 2 others: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Jdforrester-WMF) [19:33:59] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2230.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912021933_dzahn_74631_mw223... [19:40:33] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1295.eqiad.wmnet', 'mw1294.eqiad.wmnet', 'mw1293.eqiad.wmnet'] ` and were **ALL** successful. [19:47:04] ahhh i needed to use the EC (?) version of the key [20:13:01] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2245.codfw.wmnet'] ` and were **ALL** successful. [20:33:58] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2247.codfw.wmnet'] ` and were **ALL** successful. [20:35:29] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2246.codfw.wmnet'] ` and were **ALL** successful. [20:39:11] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2231.codfw.wmnet'] ` and were **ALL** successful. [20:39:16] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2233.codfw.wmnet'] ` and were **ALL** successful. [20:41:35] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2234.codfw.wmnet'] ` and were **ALL** successful. [20:42:07] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2232.codfw.wmnet'] ` and were **ALL** successful. [20:43:38] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) @akosiaris I merged and applied https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/551610 in staging.... [20:47:55] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2230.codfw.wmnet'] ` and were **ALL** successful. [21:34:42] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2248.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022134_dzahn_104033_mw22... [21:35:16] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2249.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022135_dzahn_105363_mw22... [21:36:33] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2251.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022136_dzahn_105527_mw22... [21:39:38] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10Dzahn) [22:23:00] 10serviceops: Build and supply buster nodejs (nodejs-slim and nodejs-devel) container images - https://phabricator.wikimedia.org/T238927 (10Jdforrester-WMF) [22:34:34] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2248.codfw.wmnet'] ` and were **ALL** successful. [22:43:05] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2249.codfw.wmnet'] ` and were **ALL** successful. [22:48:57] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2251.codfw.wmnet'] ` and were **ALL** successful. [22:57:13] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [22:58:00] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) https://noc.wikimedia.org has been switched to use https://mwmaint.discovery.wmnet (envoy on mwmaint1002). [23:38:04] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2250.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022337_dzahn_132438_mw22... [23:38:41] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2252.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022338_dzahn_132541_mw22... [23:42:36] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2253.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022342_dzahn_132970_mw22... [23:43:09] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2254.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201912022343_dzahn_133083_mw22...