[00:15:17] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2293.codfw.wmnet'] ` and were **ALL** successful. [00:16:24] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2294.codfw.wmnet'] ` and were **ALL** successful. [00:17:22] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2295.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [00:17:24] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2296.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [00:19:21] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) [00:39:02] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2295.codfw.wmnet'] ` and were **ALL** successful. [00:39:09] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2297.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [00:41:10] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2296.codfw.wmnet'] ` and were **ALL** successful. [00:41:16] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2298.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [01:00:06] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2297.codfw.wmnet'] ` and were **ALL** successful. [01:02:59] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2299.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [01:03:28] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2298.codfw.wmnet'] ` and were **ALL** successful. [01:03:50] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2300.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [01:24:43] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2299.codfw.wmnet'] ` and were **ALL** successful. [01:26:27] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2300.codfw.wmnet'] ` and were **ALL** successful. [02:19:08] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2301.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [02:27:55] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2302.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [02:40:05] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2301.codfw.wmnet'] ` and were **ALL** successful. [02:45:13] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2303.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [02:47:58] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2302.codfw.wmnet'] ` and were **ALL** successful. [02:49:27] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2304.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [03:07:15] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2303.codfw.wmnet'] ` and were **ALL** successful. [03:10:19] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2304.codfw.wmnet'] ` and were **ALL** successful. [03:12:48] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2305.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [03:13:03] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2306.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [03:34:44] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2305.codfw.wmnet'] ` and were **ALL** successful. [03:37:09] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2306.codfw.wmnet'] ` and were **ALL** successful. [03:37:42] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2307.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [03:37:48] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2308.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [03:37:53] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2309.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [03:56:17] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) |servers Rack A3 |OS|ready for service| |servers Rack A6 |OS|ready for service| |mw2291|Stretch|yes| |mw2301|Stretch|yes| |... [03:59:32] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2307.codfw.wmnet'] ` and were **ALL** successful. [04:01:51] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2308.codfw.wmnet'] ` and were **ALL** successful. [04:02:27] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2309.codfw.wmnet'] ` and were **ALL** successful. [07:14:11] 10serviceops, 10Operations, 10Traffic: Add x-request-id to httpd (apache) logs - https://phabricator.wikimedia.org/T244545 (10Joe) 05Open→03Resolved a:03Joe [07:52:22] 10serviceops, 10Analytics, 10Operations, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10elukey) My bad, today I reviewed irc2001 with Moritz and it turned up that a GNOME deployment happened. This is because when d-i was running an... [10:51:28] 10serviceops, 10MediaWiki-General, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team): siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Joe) To be clear, I think what @Anomie described is a perfectly valid way of handling ca... [11:44:57] 10serviceops, 10Android-app-Bugs, 10Operations, 10Traffic, and 3 others: When fetching siteinfo from the MediaWiki API, set the `maxage` and `smaxage` parameters. - https://phabricator.wikimedia.org/T245033 (10jbond) p:05Triage→03Medium [16:34:43] 10serviceops, 10MediaWiki-General, 10Operations: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10Joe) [16:35:03] 10serviceops, 10MediaWiki-General, 10Operations: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10Joe) [16:49:08] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10WDoranWMF) p:05Triage→03High [17:25:46] 10serviceops, 10Operations, 10Patch-For-Review: Test gutter pool failover in production and memcached 1.5.x - https://phabricator.wikimedia.org/T240684 (10jijiki) [17:52:33] 10serviceops, 10Operations, 10observability, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) I have uploaded a patch that could possibly work, my issue generally is that I can't find a sane and safe way to test if those logstas... [18:09:09] 10serviceops, 10Operations, 10observability, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10herron) Hey @jijiki, usually to test/validate filters like this I'll cherry pick or live-hack the logstash config on the beta cluster and gene... [18:30:56] 10serviceops, 10Operations: Opcache corruption: Object and method magically missing - https://phabricator.wikimedia.org/T245183 (10jijiki) p:05Triage→03Medium [18:39:13] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption: Method call proxied to unrelated object (also: Call to undefined method) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [18:39:27] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [18:55:58] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [19:05:55] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method) - https://phabricator.wikimedia.org/T245183 (10Krinkle) Here is another mysterious mis-call which I can't explain: [Logstash single document](https://logstash.... [19:13:02] 10serviceops, 10Operations, 10observability, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) @herron I fiddled a bit on beta, it appears that for some reason, nothing is being streamed there since today, I am not sure if I brok... [19:17:02] 10serviceops, 10Operations: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10jijiki) 05Open→03Resolved a:03jijiki thank you daniel! [19:47:18] 10serviceops, 10Operations, 10ops-eqiad: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Cmjohnson) [19:48:32] 10serviceops, 10Operations: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Cmjohnson) a:05Cmjohnson→03Joe @Joe These are ready for you now, initial puppet run has been completed. I am removing from ops-eqiad tag. If you need any additional on-sit... [20:57:25] 10serviceops, 10Analytics, 10Analytics-Kanban: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [21:46:32] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) Just applied [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/572095 | this ]] for eve... [21:57:53] hmm, httpbb is getting errors from mw on some fetches: "You do not have permission to read this page, for the following reason: Your username or IP address has been automatically blocked by MediaWiki. The reason given is: Anonymous contributions are not allowed from your IP address (10.64.32.25). Please log in." [21:57:57] (that IP address is cumin1001) [21:58:07] any hints what that's about? [22:00:04] (no attempt to edit anything, just URLs like https://wikimaniateam.wikimedia.org -- which works fine from my home IP address, but not from cumin1001) [22:08:56] <_joe_> uhm that's pretty strange, that's also a private wiki IIRC [22:09:13] yeah I wouldn't expect to be able to read pages on it, but / should work [22:10:58] <_joe_> I would guess it's some abusefilter if I had to bet [22:11:56] strictly speaking it doesn't need to be fixed -- even getting the Permission Error page is proof enough that the apache config is working [22:12:04] and it includes the name of the wiki so I can assert on that [22:12:06] it's just weird [22:12:56] <_joe_> well I'd like to understand why is that [22:14:11] yeah -- especially if httpbb triggers it by hitting a rate limit or something, that wouldn't be great [22:18:20] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/572106 | eventgate-main will change ]... [22:27:14] hmmmm, I might be hitting https://phabricator.wikimedia.org/T154698 because I'm coming from 10.0.0.0/8 -- but still not sure why it's blocking a read [22:30:34] 10serviceops, 10Operations, 10observability, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10herron) Fwiw I do see logs flowing into logstash-beta generally, but puppet was broken in the beta cluster because the master filled its disk....