[01:16:41] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:16:53] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=get https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:28:11] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=get https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:32:07] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation=list https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:36:01] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:37:49] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=get https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:40:51] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:45:55] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=get https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:46:37] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation=create https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:50:33] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:50:45] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=get https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:57:13] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=get https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:58:39] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:00:26] (03CR) 10Viztor: "Same for all three files, see zhwiki-hans.png zhwiki-hans-1.5x.png and zhwiki-hans-2x.png. It's not in the config but seems to conventiona" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 (owner: 10Viztor) [02:02:45] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation=create https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:05:15] RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:06:41] RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:37] RECOVERY - etcd request latencies on neon is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:08:27] (03PS5) 10Viztor: Add hd variations for zhwikiource project logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 [02:08:57] (03CR) 10Viztor: "> Patch Set 4:" (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 (owner: 10Viztor) [02:24:43] PROBLEM - puppet last run on analytics1056 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [02:42:19] PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 93346664 and 5 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [02:45:35] RECOVERY - Postgres Replication Lag on maps2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 148920 and 32 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [02:52:43] RECOVERY - puppet last run on analytics1056 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:15:11] PROBLEM - Juniper alarms on csw2-esams is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.21.0.105 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [03:26:43] PROBLEM - toolschecker: All k8s etcd nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/k8s - 518 bytes in 3.705 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [03:32:15] RECOVERY - toolschecker: All k8s etcd nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 1.008 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [03:32:59] RECOVERY - Juniper alarms on csw2-esams is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [03:39:29] PROBLEM - puppet last run on analytics1064 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 8 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz],File[/usr/share/GeoIP/GeoIP2-City.mmdb.test] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:41:17] PROBLEM - Juniper alarms on csw2-esams is CRITICAL: JNX_ALARMS CRITICAL - Received genError(5) error-status at error-index 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [03:44:25] RECOVERY - Juniper alarms on csw2-esams is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [03:49:27] PROBLEM - Juniper alarms on csw2-esams is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.21.0.105 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [03:59:05] RECOVERY - Juniper alarms on csw2-esams is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [04:01:53] RECOVERY - puppet last run on analytics1064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [04:04:07] PROBLEM - Juniper alarms on csw2-esams is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.21.0.105 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [04:25:05] RECOVERY - Juniper alarms on csw2-esams is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [06:28:59] PROBLEM - puppet last run on cp2024 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/puppetlabs/facter/facter.conf] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:29:07] PROBLEM - puppet last run on elastic1045 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:29:25] PROBLEM - puppet last run on ms-be1044 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:29:39] PROBLEM - puppet last run on cp2010 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/apt2xml] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:30:25] PROBLEM - puppet last run on elastic2054 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/apt2xml] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:32:24] (03PS1) 10Tulsi Bhagat: Add Namespaces translation for zh.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527862 [06:32:27] PROBLEM - puppet last run on restbase2018 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ferm/ferm.conf] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:33:17] PROBLEM - puppet last run on mw2222 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:33:17] PROBLEM - puppet last run on restbase1023 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:34:11] PROBLEM - puppet last run on deploy1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/mysql-ps1.sh] https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:37:23] (03PS2) 10Tulsi Bhagat: Add Namespaces translation for zh.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527862 (https://phabricator.wikimedia.org/T229743) [06:40:06] o/ Urbanecm, ^_^ T229743 .. Thank you! [06:40:06] T229743: Display 作者 for Author namespace on zh.wikisource - https://phabricator.wikimedia.org/T229743 [06:55:39] RECOVERY - puppet last run on mw2222 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:56:57] RECOVERY - puppet last run on cp2024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:57:07] RECOVERY - puppet last run on elastic1045 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:57:23] RECOVERY - puppet last run on ms-be1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:57:37] RECOVERY - puppet last run on cp2010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:58:25] RECOVERY - puppet last run on elastic2054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [07:00:29] RECOVERY - puppet last run on restbase2018 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [07:01:17] RECOVERY - puppet last run on restbase1023 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [07:02:07] RECOVERY - puppet last run on deploy1001 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [07:29:00] (03PS6) 10Viztor: Add hd variations for zhwikiource project logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 [07:56:19] (03CR) 10Viztor: Add redirects from 'lzh' to 'zh-classical' (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/481533 (https://phabricator.wikimedia.org/T167513) (owner: 10Fomafix) [08:43:32] 10Operations, 10Wikimedia-Mailing-lists: Yoruba Wikimedians User Group Official Mailing List - https://phabricator.wikimedia.org/T229749 (10Wikicology) [08:47:11] 10Operations, 10Wikimedia-Mailing-lists: Yoruba Wikimedians User Group Official Mailing List - https://phabricator.wikimedia.org/T229749 (10Wikicology) Looking forward to having this mailing list created [08:56:35] (03PS4) 10Fomafix: Add redirects from 'lzh' to 'zh-classical' [puppet] - 10https://gerrit.wikimedia.org/r/481533 (https://phabricator.wikimedia.org/T167513) [08:57:26] (03CR) 10Fomafix: Add redirects from 'lzh' to 'zh-classical' (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/481533 (https://phabricator.wikimedia.org/T167513) (owner: 10Fomafix) [09:14:26] (03PS4) 10Fomafix: Add 'lzh' as alias for 'zh-classical' [dns] - 10https://gerrit.wikimedia.org/r/481532 (https://phabricator.wikimedia.org/T167513) [09:16:52] (03PS5) 10Fomafix: Add redirects from 'lzh' to 'zh-classical' [puppet] - 10https://gerrit.wikimedia.org/r/481533 (https://phabricator.wikimedia.org/T167513) [10:02:38] (03PS4) 10Fomafix: Add 'sgs' as alias for 'bat-smg' [dns] - 10https://gerrit.wikimedia.org/r/481539 (https://phabricator.wikimedia.org/T204830) [10:03:22] (03PS3) 10Fomafix: Add redirects from 'sgs' to 'bat-smg' [puppet] - 10https://gerrit.wikimedia.org/r/481540 (https://phabricator.wikimedia.org/T204830) [11:22:33] 10Operations, 10netops: csw2-esams's VCP link flapped - https://phabricator.wikimedia.org/T229755 (10elukey) [12:25:37] PROBLEM - Mediawiki Cirrussearch update rate - codfw on icinga1001 is CRITICAL: CRITICAL: 30.00% of data under the critical threshold [50.0] https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [12:25:51] PROBLEM - Mediawiki Cirrussearch update rate - eqiad on icinga1001 is CRITICAL: CRITICAL: 30.00% of data under the critical threshold [50.0] https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [12:26:57] 10Operations, 10Wikimedia-Mailing-lists: Yoruba Wikimedians User Group Official Mailing List - https://phabricator.wikimedia.org/T229749 (10MarcoAurelio) [12:29:56] 10Operations, 10Wikimedia-Mailing-lists: Yoruba Wikimedians User Group Official Mailing List - https://phabricator.wikimedia.org/T229749 (10MarcoAurelio) Please note that all mailing lists are created public, and with public archives, unless stated/requested otherwise. As such: - Shall the list be public or p... [12:41:45] RECOVERY - Mediawiki Cirrussearch update rate - codfw on icinga1001 is OK: OK: Less than 1.00% under the threshold [80.0] https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [12:42:01] RECOVERY - Mediawiki Cirrussearch update rate - eqiad on icinga1001 is OK: OK: Less than 1.00% under the threshold [80.0] https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [12:54:05] PROBLEM - puppet last run on analytics1070 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [12:56:09] happy to help Tulsi! [13:17:57] (03CR) 10Urbanecm: [C: 04-1] "> Patch Set 4:" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 (owner: 10Viztor) [13:18:01] (03CR) 10Urbanecm: [C: 04-1] "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 (owner: 10Viztor) [13:22:07] RECOVERY - puppet last run on analytics1070 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [13:51:44] 10Operations, 10Wikimedia-Mailing-lists: Yoruba Wikimedians User Group Official Mailing List - https://phabricator.wikimedia.org/T229749 (10Wikicology) *The list should be private and requires list administrators approval for subscription. *The list should have archive and should be private [16:02:51] PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga1001 is CRITICAL: cluster=cache_text site=esams https://wikitech.wikimedia.org/wiki/Cache_TLS_termination https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [16:07:39] RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Cache_TLS_termination https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [16:10:16] (03CR) 10Urbanecm: "> Patch Set 2: Code-Review+1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527632 (https://phabricator.wikimedia.org/T229644) (owner: 10Isarra) [17:55:56] (03CR) 10Urbanecm: [C: 04-2] "Task says "Maybe there has been a miscommunication, in this case, the namespace itself (in the url) should not change, instead, it should " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527862 (https://phabricator.wikimedia.org/T229743) (owner: 10Tulsi Bhagat) [17:57:58] (03Abandoned) 10Urbanecm: Remove NS 104 from wgContentNamespaces for euwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481266 (https://phabricator.wikimedia.org/T191396) (owner: 10Framawiki) [18:11:58] (03PS1) 10Daimona Eaytoy: Use AbuseFilterParser on beta enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527906 [18:13:44] (03PS2) 10Daimona Eaytoy: Use AbuseFilterParser on beta enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527906 [18:15:10] (03PS7) 10Viztor: Add hd variations for zhwikiource project logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527773 [18:31:07] (03PS1) 10Fomafix: Add 'nrf' as alias for 'nrm' [dns] - 10https://gerrit.wikimedia.org/r/527908 (https://phabricator.wikimedia.org/T25216) [18:33:37] (03PS1) 10Fomafix: Add 'nrf' as alias for 'nrm' [puppet] - 10https://gerrit.wikimedia.org/r/527909 (https://phabricator.wikimedia.org/T25216) [18:38:32] (03CR) 10Krinkle: [C: 03+1] "Sounds good, OK to deploy for labs now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527906 (owner: 10Daimona Eaytoy) [18:38:43] (03CR) 10Krinkle: [C: 03+2] Use AbuseFilterParser on beta enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527906 (owner: 10Daimona Eaytoy) [18:39:45] (03Merged) 10jenkins-bot: Use AbuseFilterParser on beta enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527906 (owner: 10Daimona Eaytoy) [18:40:02] (03CR) 10jenkins-bot: Use AbuseFilterParser on beta enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527906 (owner: 10Daimona Eaytoy) [18:45:07] !log krinkle@deploy1001 Synchronized wmf-config/abusefilter.php: labs-only noop - f740f89c594979 (duration: 00m 50s) [18:45:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:04:08] (03PS1) 10Fomafix: Add 'cbk' as alias for 'cbk-zam' [dns] - 10https://gerrit.wikimedia.org/r/527911 (https://phabricator.wikimedia.org/T124657) [19:04:24] (03PS1) 10Fomafix: Add 'cbk' as alias for 'cbk-zam' [puppet] - 10https://gerrit.wikimedia.org/r/527912 (https://phabricator.wikimedia.org/T124657) [19:35:43] (03PS1) 10Fomafix: Add 'vro' as alias for 'fiu-vro' [dns] - 10https://gerrit.wikimedia.org/r/527914 (https://phabricator.wikimedia.org/T31186) [19:36:11] (03PS1) 10Fomafix: Add 'vro' as alias for 'fiu-vro' [puppet] - 10https://gerrit.wikimedia.org/r/527915 (https://phabricator.wikimedia.org/T31186) [19:51:18] (03PS1) 10Fomafix: Add 'rup' as alias for 'roa-rup' [dns] - 10https://gerrit.wikimedia.org/r/527916 (https://phabricator.wikimedia.org/T17988) [19:51:30] (03PS1) 10Fomafix: Add 'rup' as alias for 'roa-rup' [puppet] - 10https://gerrit.wikimedia.org/r/527917 (https://phabricator.wikimedia.org/T17988) [20:01:11] PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: /{domain}/v1/transform/html/to/mobile-html/{title} (Get preview mobile HTML for test page) is CRITICAL: Test Get preview mobile HTML for test page returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [20:02:49] RECOVERY - mobileapps endpoints health on scb2003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [20:12:29] PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: /{domain}/v1/page/metadata/{title} (retrieve extended metadata for Video article on English Wikipedia) is CRITICAL: Test retrieve extended metadata for Video article on English Wikipedia returned the unexpected status 504 (expecting: 200): /{domain}/v1/page/mobile-html/{title} (Get page content HTML for test page) is CRITICAL: Test Get page content HTML for test page r [20:12:29] ected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [20:14:07] RECOVERY - mobileapps endpoints health on scb2003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [20:49:41] (03PS1) 10Viztor: Update HD logo for en.ws and mul.ws [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527922 (https://phabricator.wikimedia.org/T211374) [20:53:19] (03PS2) 10Viztor: Update HD logo for en.ws and mul.ws [mediawiki-config] - 10https://gerrit.wikimedia.org/r/527922 (https://phabricator.wikimedia.org/T229769) [21:19:10] (03PS1) 10Fomafix: Add 'egl' as alias for 'eml' [dns] - 10https://gerrit.wikimedia.org/r/527932 (https://phabricator.wikimedia.org/T36217) [21:19:22] (03PS1) 10Fomafix: Add 'egl' as alias for 'eml' [puppet] - 10https://gerrit.wikimedia.org/r/527933 (https://phabricator.wikimedia.org/T36217) [21:46:41] PROBLEM - puppet last run on cloudnet1003 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [22:14:43] RECOVERY - puppet last run on cloudnet1003 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [23:08:39] PROBLEM - mobileapps endpoints health on scb2005 is CRITICAL: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most read articles for January 1, 2016) is CRITICAL: Test retrieve the most read articles for January 1, 2016 returned the unexpected status 504 (expecting: 200): /{domain}/v1/data/css/mobile/site (Get site-specific CSS) is CRITICAL: Test Get site-specific CSS returned the unexpected status 504 (expecting: [23:08:39] 1/page/random/title (retrieve a random article title) is CRITICAL: Test retrieve a random article title returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [23:18:27] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/transform/html/to/mobile-html/{title} (Get preview mobile HTML for test page) is CRITICAL: Test Get preview mobile HTML for test page returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [23:20:03] RECOVERY - mobileapps endpoints health on scb2005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [23:21:41] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [23:24:53] PROBLEM - mobileapps endpoints health on scb2005 is CRITICAL: /{domain}/v1/page/media/{title} (Get media in test page) is CRITICAL: Test Get media in test page returned the unexpected status 504 (expecting: 200): /{domain}/v1/page/mobile-html/{title} (Get page content HTML for test page) is CRITICAL: Test Get page content HTML for test page returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Se [23:24:53] /mobileapps [23:28:07] RECOVERY - mobileapps endpoints health on scb2005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [23:39:33] PROBLEM - mobileapps endpoints health on scb2005 is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) is CRITICAL: Test Get site-specific CSS returned the unexpected status 504 (expecting: 200): /{domain}/v1/page/summary/{title} (Get summary for test page) is CRITICAL: Test Get summary for test page returned the unexpected status 504 (expecting: 200): /{domain}/v1/page/featured/{year}/{month}/{day} (retrieve tit [23:39:33] d article for April 29, 2016) is CRITICAL: Test retrieve title of the featured article for April 29, 2016 returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [23:42:53] RECOVERY - mobileapps endpoints health on scb2005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps