[00:11:55] (03CR) 10Bstorm: "*grumbles something about the heaviness of vagrant*" (031 comment) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578166 (owner: 10BryanDavis) [00:15:44] (03CR) 10BryanDavis: rebuild_all: Allow overriding python used and additional args (031 comment) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578166 (owner: 10BryanDavis) [00:16:27] (03CR) 10Bstorm: [C: 03+2] "Ah ok, I get it." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578166 (owner: 10BryanDavis) [00:17:25] (03Merged) 10jenkins-bot: rebuild_all: Allow overriding python used and additional args [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578166 (owner: 10BryanDavis) [00:21:34] (03CR) 10Bstorm: [C: 03+2] "This I was able to test in my local environment just using Docker. It ran well :)" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578167 (owner: 10BryanDavis) [00:22:14] (03Merged) 10jenkins-bot: Introduce jinja2 macros [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578167 (owner: 10BryanDavis) [00:32:54] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31899 bytes in 0.006 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [00:39:24] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [00:43:14] (03CR) 10Bstorm: [C: 04-1] "This worked well in toolsbeta with a live patch on puppet and the script referenced in the task. I feel quite confident this will do what " [puppet] - 10https://gerrit.wikimedia.org/r/584061 (https://phabricator.wikimedia.org/T248702) (owner: 10Bstorm) [01:09:56] (03CR) 10Reedy: gerrit::proxy: Switch to strong SSL settings (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/505410 (https://phabricator.wikimedia.org/T221499) (owner: 10Alex Monk) [01:10:43] 10Operations, 10Gerrit, 10Traffic, 10HTTPS, and 2 others: Disable TLS 1.0 and 1.1 in apache for gerrit.wikimedia.org - https://phabricator.wikimedia.org/T221499 (10Reedy) Looks like this has been done {F31706596 size=full} Do we still want to change cipher stuff? Should we move that to another task? [02:18:58] (03PS2) 10Reedy: gerrit::proxy: Switch to strong SSL settings [puppet] - 10https://gerrit.wikimedia.org/r/505410 (https://phabricator.wikimedia.org/T221499) (owner: 10Alex Monk) [02:30:07] (03PS1) 10Reedy: Add export-11 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584095 (https://phabricator.wikimedia.org/T238921) [03:23:00] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31897 bytes in 8.297 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [03:31:32] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:17:46] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31897 bytes in 2.737 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:26:14] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:49:20] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31897 bytes in 0.007 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:57:54] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [05:37:56] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31897 bytes in 0.007 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [05:42:17] 10Operations, 10Traffic, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Ammarpad) [05:50:44] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [05:50:54] (03CR) 10Hashar: [C: 03+1] "The suites are defined in modules/wmflib/lib/puppet/parser/functions/ssl_ciphersuite.rb , 'mid' is a superset of 'strong' with the additi" [puppet] - 10https://gerrit.wikimedia.org/r/505410 (https://phabricator.wikimedia.org/T221499) (owner: 10Alex Monk) [05:51:40] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1089 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer https://wikitech.wikimedia.org/wiki/HTTPS [05:51:40] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1089 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer https://wikitech.wikimedia.org/wiki/HTTPS [05:52:23] 10Operations, 10Gerrit, 10Traffic, 10HTTPS, and 2 others: Disable TLS 1.0 and 1.1 in apache for gerrit.wikimedia.org - https://phabricator.wikimedia.org/T221499 (10hashar) The old TLS has been disabled by the traffic team. @Krenair patch propose to elevate the set of cypher to a stronger set which seems go... [06:19:48] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1089 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer https://wikitech.wikimedia.org/wiki/HTTPS [06:32:46] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1089 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer https://wikitech.wikimedia.org/wiki/HTTPS [06:54:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1089 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer https://wikitech.wikimedia.org/wiki/HTTPS [08:16:31] (03PS1) 10Zhuyifei1999: toolforge/mailrelay: Fix wrong source URI [puppet] - 10https://gerrit.wikimedia.org/r/584102 (https://phabricator.wikimedia.org/T248731) [08:30:45] !log restarting ats-tls on cp1089 [08:30:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:31:00] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31690 bytes in 0.008 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [08:31:36] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1089 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 293305 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2020-06-13 17:01:29 +0000 (expires in 77 days) https://wikitech.wikimedia.org/wiki/HTTPS [08:32:32] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1089 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 293249 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2020-06-13 17:01:21 +0000 (expires in 77 days) https://wikitech.wikimedia.org/wiki/HTTPS [08:34:28] !log pool cp1089 [08:34:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:39:28] (03CR) 10Elukey: "Forgot to review yesterday sorry!" [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) (owner: 10Mstyles) [08:40:07] 10Operations, 10Traffic: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 (10Vgutierrez) [08:44:00] 10Operations, 10Traffic: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 (10Vgutierrez) cp1089 [[ https://grafana.wikimedia.org/d/6uhkG6OZk/ats-instance-drilldown?orgId=1&var-site=eqiad%20prometheus%2Fops&var-instance=cp1089&var-layer=tls&from=1584780221656&to=1585385021657 | shows... [08:54:18] 10Operations, 10Traffic: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 (10Vgutierrez) [[ https://grafana.wikimedia.org/d/6uhkG6OZk/ats-instance-drilldown?orgId=1&from=1584780826363&to=1585385626363&var-site=eqiad%20prometheus%2Fops&var-instance=cp1089&var-layer=tls | even though... [08:54:35] 10Operations, 10Traffic: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 (10Vgutierrez) p:05Triage→03Medium [08:58:34] 10Operations, 10Traffic: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 (10Vgutierrez) @ema we need to check if this can be related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/583295 [10:11:02] 10Operations, 10Gerrit, 10Traffic, 10HTTPS, and 2 others: Disable TLS 1.0 and 1.1 in apache for gerrit.wikimedia.org - https://phabricator.wikimedia.org/T221499 (10Reedy) >>! In T221499#6006826, @hashar wrote: > The old TLS has been disabled by the traffic team. @Krenair patch propose to elevate the set of... [10:47:00] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1077 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [10:53:08] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1077 is OK: HTTP OK: HTTP/1.1 200 Ok - 31888 bytes in 4.323 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [11:01:40] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1077 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [11:32:31] !log restart ats-tls on cp1077 - T248736 [11:32:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:32:38] T248736: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 [11:37:26] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1077 is OK: HTTP OK: HTTP/1.1 200 Ok - 31760 bytes in 0.009 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [12:01:27] (03PS1) 10Vgutierrez: Revert "ATS: disable transaction_active_timeout_in for EventStreams" [puppet] - 10https://gerrit.wikimedia.org/r/584110 [12:03:28] (03PS2) 10Vgutierrez: Revert "ATS: disable transaction_active_timeout_in for EventStreams" [puppet] - 10https://gerrit.wikimedia.org/r/584110 (https://phabricator.wikimedia.org/T248736) [12:05:29] !log preemptive restart of ats-tls on cp1081 and cp3062 - T248736 [12:05:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:05:35] T248736: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 [12:29:38] (03CR) 10Vgutierrez: [C: 03+2] "+1ed by ema offline" [puppet] - 10https://gerrit.wikimedia.org/r/584110 (https://phabricator.wikimedia.org/T248736) (owner: 10Vgutierrez) [12:32:16] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is CRITICAL: 125.1 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1003&panelId=37 [12:35:34] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is CRITICAL: 107.8 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1004&panelId=37 [12:40:38] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 111.9 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [12:48:26] (03CR) 10L0st3xpl0r3r: "> Patch Set 9: Verified+2" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) (owner: 10L0st3xpl0r3r) [12:50:35] (03CR) 10L0st3xpl0r3r: "> Patch Set 9:" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) (owner: 10L0st3xpl0r3r) [15:22:14] PROBLEM - Hadoop NodeManager on analytics1071 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [15:48:02] (03CR) 10Cwhite: [C: 03+1] "As others have said, more description of what and why in the commit message is helpful. Otherwise from a cursory glance it looks good to " [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) (owner: 10Mstyles) [15:50:42] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 79.32 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [15:54:50] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is OK: (C)100 gt (W)80 gt 76.55 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1003&panelId=37 [16:16:52] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is OK: (C)100 gt (W)80 gt 76.55 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1004&panelId=37 [16:49:04] (03PS1) 10Urbanecm: Add collections.nmnh.si.edu to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584126 (https://phabricator.wikimedia.org/T248659) [16:49:48] (03CR) 10jerkins-bot: [V: 04-1] Add collections.nmnh.si.edu to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584126 (https://phabricator.wikimedia.org/T248659) (owner: 10Urbanecm) [16:50:53] (03PS2) 10Urbanecm: Add collections.nmnh.si.edu to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584126 (https://phabricator.wikimedia.org/T248659) [16:54:36] !log restart yarn on analytics1071 [16:54:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:55:56] RECOVERY - Hadoop NodeManager on analytics1071 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [16:57:22] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 101.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [17:16:16] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 79.32 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [17:25:20] PROBLEM - Debian mirror in sync with upstream on sodium is CRITICAL: /srv/mirrors/debian is over 14 hours old. https://wikitech.wikimedia.org/wiki/Mirrors [17:50:44] 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, 10Traffic, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10CCicalese_WMF) [17:53:01] 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, 10Traffic, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10CCicalese_WMF) @Krinkle Are there any more items mentioned in T203899#4895417 that need to be done? Is the item above about purging j... [17:56:08] (03PS1) 10Gergő Tisza: Alphabetize GrowthExperiments settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584132 [17:56:10] (03PS1) 10Gergő Tisza: Deploy GrowthExperiments on Serbian Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584133 (https://phabricator.wikimedia.org/T241181) [18:04:54] (03PS1) 10Gergő Tisza: Enable GrowthExperiments welcome survey on Ukrainian, Hungarian, Armenian Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584135 (https://phabricator.wikimedia.org/T238295) [18:36:02] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 107.8 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [18:41:02] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3050 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [18:42:58] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3050 is OK: HTTP OK: HTTP/1.0 200 OK - 22376 bytes in 0.256 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [19:25:46] 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, 10Traffic, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Krinkle) * JobQueue: I defer to CPT. I'm not sure how to check that. * T203899#4895417: The only thing mentioned there not yet addres... [19:26:24] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 79.32 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [20:43:46] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 108.8 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [20:48:00] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is CRITICAL: 101.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1003&panelId=37 [21:21:26] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 78.31 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [21:25:40] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is OK: (C)100 gt (W)80 gt 75.25 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1003&panelId=37 [21:34:42] (03PS1) 10RhinosF1: Adding additional namespaces to tr.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 [21:41:27] (03PS2) 10RhinosF1: Adding additional namespaces to tr.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 (https://phabricator.wikimedia.org/T248734) [21:42:29] (03CR) 10jerkins-bot: [V: 04-1] Adding additional namespaces to tr.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 (https://phabricator.wikimedia.org/T248734) (owner: 10RhinosF1) [21:43:50] (03PS3) 10RhinosF1: Adding additional namespaces to tr.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 (https://phabricator.wikimedia.org/T248734) [21:45:34] CI exists to -1 me, fixed. [22:01:54] (03PS4) 10RhinosF1: Add 3 additional namespaces and assoicated talk pages to trwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 (https://phabricator.wikimedia.org/T248734) [22:02:52] (03CR) 10jerkins-bot: [V: 04-1] Add 3 additional namespaces and assoicated talk pages to trwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 (https://phabricator.wikimedia.org/T248734) (owner: 10RhinosF1) [22:03:54] ^ has Jenkins gone mad, fatal: Could not read from remote repository. [22:05:58] (03CR) 10RhinosF1: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584172 (https://phabricator.wikimedia.org/T248734) (owner: 10RhinosF1) [22:51:59] (03PS2) 10Gergő Tisza: Alphabetize GrowthExperiments settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584132 [22:52:01] (03PS2) 10Gergő Tisza: Deploy GrowthExperiments on Serbian Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584133 (https://phabricator.wikimedia.org/T241181) [22:52:03] (03PS2) 10Gergő Tisza: Enable GrowthExperiments welcome survey on Ukrainian, Hungarian, Armenian Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584135 (https://phabricator.wikimedia.org/T238295) [22:52:06] (03PS1) 10Gergő Tisza: Enable GrowthExperiments on French Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584183 (https://phabricator.wikimedia.org/T235964) [23:04:04] (03PS3) 10Gergő Tisza: Deploy GrowthExperiments on Serbian Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584133 (https://phabricator.wikimedia.org/T241181) [23:04:06] (03PS3) 10Gergő Tisza: Enable GrowthExperiments welcome survey on Ukrainian, Hungarian, Armenian Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584135 (https://phabricator.wikimedia.org/T238295) [23:04:08] (03PS2) 10Gergő Tisza: Enable GrowthExperiments on French Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584183 (https://phabricator.wikimedia.org/T235964) [23:04:10] (03PS1) 10Gergő Tisza: Remove no-op GrowthExperiments beta settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/584184