[00:47:18] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1329.eqiad.wmnet'] ` and were **ALL** successful. [07:26:30] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1312.eqiad.wmnet', 'mw1308.eqiad.wmnet', 'mw1261.eqiad.wmnet'] ` The log can be found in `/var/log/... [07:26:34] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1262.eqiad.wmnet', 'mw1278.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [07:46:16] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [07:46:43] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [08:19:18] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw2223.codfw.wmnet', 'mw2222.codfw.wmnet', 'mw2221.codfw.wmnet', 'mw2220.codfw.wmnet'] ` The log can... [09:18:36] 10serviceops, 10Operations: Kubernetes emits log level 50 - https://phabricator.wikimedia.org/T239459 (10jijiki) [09:24:20] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1263.eqiad.wmnet', 'mw1307.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [09:41:54] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1278.eqiad.wmnet'] ` Of which those **FAILED**: ` ['mw1278.eqiad.wmnet'] ` [10:22:54] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1306.eqiad.wmnet', 'mw1264.eqiad.wmnet', 'mw1279.eqiad.wmnet'] ` The log can be found in `/var/log/... [10:27:00] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) >>! In T239054#5701191, @ops-monitoring-bot wrote: > Completed auto-reimage of hosts: > ` > ['mw1278.eqiad.wmnet'] > ` > > Of which those **FAILED**: > ` > ['mw1278.eqiad.wmnet'] > ` I checked the... [10:44:27] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1308.eqiad.wmnet'] ` Of which those **FAILED**: ` ['mw1308.eqiad.wmnet'] ` [10:58:36] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1268.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw1281.eqiad.wmnet'] ` The log can be found in `/var/log/... [11:39:16] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1263.eqiad.wmnet', 'mw1307.eqiad.wmnet'] ` and were **ALL** successful. [11:59:32] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1305.eqiad.wmnet', 'mw1265.eqiad.wmnet', 'mw1270.eqiad.wmnet'] ` The log can be found in `/var/log/... [13:20:11] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2223.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2223.codfw.wmnet'] ` [13:44:17] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) >>! In T239054#5701280, @ops-monitoring-bot wrote: > Completed auto-reimage of hosts: > ` > ['mw1308.eqiad.wmnet'] > ` > > Of which those **FAILED**: > ` > ['mw1308.eqiad.wmnet'] > ` I run and che... [13:44:32] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1306.eqiad.wmnet', 'mw1264.eqiad.wmnet', 'mw1279.eqiad.wmnet'] ` and were **ALL** successful. [14:02:25] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1271.eqiad.wmnet', 'mw1272.eqiad.wmnet', 'mw1304.eqiad.wmnet'] ` The log can be found in `/var/log/... [14:32:02] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1268.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw1281.eqiad.wmnet'] ` and were **ALL** successful. [14:37:27] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1323.eqiad.wmnet', 'mw1297.eqiad.wmnet', 'mw1273.eqiad.wmnet'] ` The log can be found in `/var/log/... [14:44:38] 10serviceops, 10Operations, 10Pybal, 10SRE-tools, 10Traffic: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10akosiaris) `need to be able to understand the pooled status` I have to question this. Why... [14:46:03] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1282.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911291445_jiji_220572.log`. [15:02:20] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by filippo on cumin1001.eqiad.wmnet for hosts: ` mw2228.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911291502_filippo_222670_... [15:02:23] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2228.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2228.codfw.wmnet'] ` [15:02:30] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by filippo on cumin1001.eqiad.wmnet for hosts: ` mw2228.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911291502_filippo_222681_... [15:24:24] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1305.eqiad.wmnet', 'mw1265.eqiad.wmnet', 'mw1270.eqiad.wmnet'] ` and were **ALL** successful. [15:39:37] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [15:58:50] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1282.eqiad.wmnet'] ` and were **ALL** successful. [16:14:06] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [16:15:30] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2228.codfw.wmnet'] ` and were **ALL** successful. [16:17:39] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1274.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911291617_jiji_242202.log`. [17:29:59] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1271.eqiad.wmnet', 'mw1272.eqiad.wmnet', 'mw1304.eqiad.wmnet'] ` and were **ALL** successful. [17:31:06] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1274.eqiad.wmnet'] ` and were **ALL** successful. [18:15:02] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1323.eqiad.wmnet', 'mw1297.eqiad.wmnet', 'mw1273.eqiad.wmnet'] ` and were **ALL** successful. [19:20:39] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1303.eqiad.wmnet', 'mw1283.eqiad.wmnet', 'mw1284.eqiad.wmnet'] ` The log can be found in `/var/log/... [19:50:56] !log reimage mw1285.eqiad.wmnet [19:51:18] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1285.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911291950_jiji_24084.log`. [20:28:21] 10serviceops, 10MediaWiki-Documentation, 10Release-Engineering-Team, 10Upstream, 10doxygen: Upgrade Doxygen (to enable INHERIT_DOCS for methods from parent classes) - https://phabricator.wikimedia.org/T152478 (10hashar) [20:29:54] 10serviceops, 10MediaWiki-Documentation, 10Release-Engineering-Team, 10Upstream, 10doxygen: Upgrade Doxygen (to enable INHERIT_DOCS for methods from parent classes) - https://phabricator.wikimedia.org/T152478 (10hashar) [21:02:55] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1285.eqiad.wmnet'] ` and were **ALL** successful. [21:12:50] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1302.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911292112_jiji_41316.log`. [22:19:27] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1302.eqiad.wmnet'] ` and were **ALL** successful. [22:45:17] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1303.eqiad.wmnet', 'mw1283.eqiad.wmnet', 'mw1284.eqiad.wmnet'] ` and were **ALL** successful.