[00:10:32] 10Operations, 10ops-eqiad: Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T230289 (10ops-monitoring-bot) [00:25:42] PROBLEM - mobileapps endpoints health on scb2004 is CRITICAL: /{domain}/v1/page/random/title (retrieve a random article title) is CRITICAL: Test retrieve a random article title returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [00:27:18] RECOVERY - mobileapps endpoints health on scb2004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [00:40:08] PROBLEM - mobileapps endpoints health on scb2004 is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) is CRITICAL: Test Get site-specific CSS returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [00:41:46] RECOVERY - mobileapps endpoints health on scb2004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [02:17:09] 10Operations, 10ops-eqiad, 10cloud-services-team: Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T230289 (10Peachey88) [02:25:05] 10Operations, 10ops-codfw, 10media-storage: Degraded RAID on ms-be2021 - https://phabricator.wikimedia.org/T230275 (10Peachey88) [03:34:36] PROBLEM - puppet last run on analytics1055 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-ISP.mmdb.gz] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:35:12] PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-ISP.mmdb.gz] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [04:02:32] RECOVERY - puppet last run on analytics1055 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [04:03:04] RECOVERY - puppet last run on mw2172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [04:05:50] PROBLEM - snapshot of s6 in codfw on db1115 is CRITICAL: snapshot for s6 at codfw taken more than 4 days ago: Most recent backup 2019-08-07 03:34:01 https://wikitech.wikimedia.org/wiki/MariaDB/Backups [08:27:09] (03PS1) 10Alex Monk: nrpe::monitor_service: Make notes_url optional for ensure=absent [puppet] - 10https://gerrit.wikimedia.org/r/529590 [08:34:40] 10Operations, 10Domains, 10Product-Design-Strategy, 10Traffic: Add a repo reference to Design Strategy web address - https://phabricator.wikimedia.org/T230053 (10Volker_E) Thanks @Dzahn! There's no other activity needed on our site, besides merging? The cron job(?) auto-updates merged patches every 15-30 m... [08:47:05] (03CR) 10Alex Monk: [C: 04-1] nrpe::monitor_service: Make notes_url optional for ensure=absent [puppet] - 10https://gerrit.wikimedia.org/r/529590 (owner: 10Alex Monk) [08:51:10] (03PS2) 10Alex Monk: nrpe::monitor_service: Make notes_url optional for ensure=absent [puppet] - 10https://gerrit.wikimedia.org/r/529590 [08:55:12] (03PS3) 10Alex Monk: nrpe::monitor_service: Make notes_url optional for ensure=absent [puppet] - 10https://gerrit.wikimedia.org/r/529590 [13:52:23] (03PS1) 10Zoranzoki21: Add Portal namespace on zhwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) [13:56:59] (03CR) 10Viztor: [C: 04-1] "See inline comment" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) (owner: 10Zoranzoki21) [14:00:07] (03CR) 1094rain: "Should be set in wgExtraNamespaces instead of wgNamespaceAliases" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) (owner: 10Zoranzoki21) [14:05:34] (03CR) 10Urbanecm: [C: 04-1] "Viztor is right :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) (owner: 10Zoranzoki21) [16:35:08] 10Operations, 10Domains, 10Product-Design-Strategy, 10Traffic: Add a repo reference to Design Strategy web address - https://phabricator.wikimedia.org/T230053 (10Dzahn) @Volker_E Yes, nothing else needed besides merging content. Each time puppet-agent runs it will try to git clone and because the code says... [20:11:52] PROBLEM - Device not healthy -SMART- on helium is CRITICAL: cluster=misc device=megaraid,18 instance=helium:9100 job=node site=eqiad https://wikitech.wikimedia.org/wiki/SMART%23Alerts https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=helium&var-datasource=eqiad+prometheus/ops [22:04:48] (03CR) 10Viztor: "> Patch Set 2:" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529175 (owner: 10Viztor) [22:07:08] (03PS2) 10Zoranzoki21: Add Portal namespace on zhwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) [22:07:17] (03CR) 10Zoranzoki21: "Fixed!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) (owner: 10Zoranzoki21) [22:24:23] (03CR) 10Urbanecm: [C: 03+1] "Looks good to me now!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529600 (https://phabricator.wikimedia.org/T230294) (owner: 10Zoranzoki21) [22:38:03] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Temporary make account creation limits more restrictive (T230304) (duration: 00m 50s) [22:38:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:38:33] (03PS1) 10Urbanecm: Temporary make account creation limits more restrictive [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529621 (https://phabricator.wikimedia.org/T230304) [22:39:42] (03CR) 10Urbanecm: [C: 03+2] Temporary make account creation limits more restrictive [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529621 (https://phabricator.wikimedia.org/T230304) (owner: 10Urbanecm) [22:40:52] (03Merged) 10jenkins-bot: Temporary make account creation limits more restrictive [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529621 (https://phabricator.wikimedia.org/T230304) (owner: 10Urbanecm) [22:41:07] (03CR) 10jenkins-bot: Temporary make account creation limits more restrictive [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529621 (https://phabricator.wikimedia.org/T230304) (owner: 10Urbanecm) [23:06:38] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/feed/availability (Retrieve feed content availability from \wikipedia.org\) timed out before a response was received https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [23:08:06] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29