[00:59:34] ACKNOWLEDGEMENT - MD RAID on restbase-dev1006 is CRITICAL: CRITICAL: State: degraded, Active: 11, Working: 11, Failed: 1, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T227394 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [00:59:37] 10Operations, 10ops-eqiad: Degraded RAID on restbase-dev1006 - https://phabricator.wikimedia.org/T227394 (10ops-monitoring-bot) [01:05:11] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=LIST https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:05:49] PROBLEM - etcd request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 operation={compareAndSwap,get,list} https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:05:53] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PATCH https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:06:09] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation={compareAndSwap,create,get,list} https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:12:39] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:13:25] PROBLEM - kubelet operational latencies on kubernetes1004 is CRITICAL: instance=kubernetes1004.eqiad.wmnet https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [01:17:51] RECOVERY - kubelet operational latencies on kubernetes1004 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [01:24:23] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=PUT https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:28:43] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=LIST https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:03:39] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=LIST https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:07] RECOVERY - etcd request latencies on chlorine is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:11] RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:29] RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:59] RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:16:11] PROBLEM - HHVM rendering on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [02:17:29] RECOVERY - HHVM rendering on mw1342 is OK: HTTP OK: HTTP/1.1 200 OK - 77222 bytes in 0.151 second response time https://wikitech.wikimedia.org/wiki/Application_servers [02:19:53] PROBLEM - Check for gridmaster host resolution UDP on cloudservices1003 is CRITICAL: DNS CRITICAL - 8.016 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [02:22:51] PROBLEM - Check for gridmaster host resolution UDP on cloudservices1003 is CRITICAL: DNS CRITICAL - 8.019 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [02:24:09] RECOVERY - Check for gridmaster host resolution UDP on cloudservices1003 is OK: DNS OK - 0.015 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [02:35:19] PROBLEM - Check for gridmaster host resolution TCP on cloudservices1003 is CRITICAL: DNS CRITICAL - 5.583 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [02:36:41] RECOVERY - Check for gridmaster host resolution TCP on cloudservices1003 is OK: DNS OK - 0.016 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [02:56:31] PROBLEM - Check for gridmaster host resolution UDP on cloudservices1003 is CRITICAL: DNS CRITICAL - 8.119 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [02:57:51] RECOVERY - Check for gridmaster host resolution UDP on cloudservices1003 is OK: DNS OK - 0.015 seconds response time (tools-sgegrid-master.tools.eqiad.wmflabs. 60 IN A 172.16.4.197) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [03:00:01] PROBLEM - Mediawiki Cirrussearch update rate - eqiad on icinga1001 is CRITICAL: CRITICAL: 20.00% of data under the critical threshold [50.0] https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [03:00:07] PROBLEM - Mediawiki Cirrussearch update rate - codfw on icinga1001 is CRITICAL: CRITICAL: 20.00% of data under the critical threshold [50.0] https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [03:27:35] RECOVERY - Mediawiki Cirrussearch update rate - eqiad on icinga1001 is OK: OK: Less than 1.00% under the threshold [80.0] https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [03:27:43] RECOVERY - Mediawiki Cirrussearch update rate - codfw on icinga1001 is OK: OK: Less than 1.00% under the threshold [80.0] https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44&fullscreen&orgId=1 [03:28:15] PROBLEM - snapshot of s6 in codfw on db1115 is CRITICAL: snapshot for s6 at codfw taken more than 4 days ago: Most recent backup 2019-07-03 03:24:23 https://wikitech.wikimedia.org/wiki/MariaDB/Backups [03:47:49] PROBLEM - toolschecker: All Flannel etcd nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/flannel - 522 bytes in 3.026 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [03:53:15] RECOVERY - toolschecker: All Flannel etcd nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 1.496 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [03:58:31] PROBLEM - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.119 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [05:46:07] PROBLEM - snapshot of s3 in codfw on db1115 is CRITICAL: snapshot for s3 at codfw taken more than 4 days ago: Most recent backup 2019-07-03 05:20:56 https://wikitech.wikimedia.org/wiki/MariaDB/Backups [06:42:48] (03CR) 10Volans: "Thanks for migrating it! A couple of minor things inline." (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/520290 (owner: 10Jbond) [06:57:57] PROBLEM - puppet last run on bast4002 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. [07:25:11] RECOVERY - puppet last run on bast4002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:52:53] PROBLEM - HHVM rendering on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [07:54:11] RECOVERY - HHVM rendering on mw1233 is OK: HTTP OK: HTTP/1.1 200 OK - 77222 bytes in 0.155 second response time https://wikitech.wikimedia.org/wiki/Application_servers [08:40:53] PROBLEM - Host restbase2009 is DOWN: PING CRITICAL - Packet loss = 100% [08:41:49] PROBLEM - Restbase LVS codfw on restbase.svc.codfw.wmnet is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase [08:41:49] PROBLEM - restbase endpoints health on restbase2018 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:42:47] PROBLEM - restbase endpoints health on restbase2012 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:42:53] PROBLEM - restbase endpoints health on restbase2013 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:43:11] PROBLEM - restbase endpoints health on restbase2017 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:43:11] PROBLEM - restbase endpoints health on restbase2020 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:43:13] RECOVERY - restbase endpoints health on restbase2018 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:44:07] RECOVERY - restbase endpoints health on restbase2012 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:44:13] RECOVERY - restbase endpoints health on restbase2013 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:44:35] RECOVERY - restbase endpoints health on restbase2020 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:44:35] RECOVERY - restbase endpoints health on restbase2017 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [08:47:31] RECOVERY - Restbase LVS codfw on restbase.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [08:57:39] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10RhinosF1) [08:58:08] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10RhinosF1) p:05Triage→03Unbreak! Site is down - if this isn't planned then it should probably be UBN! [09:00:54] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10Varnent) This was indeed planned. It will return once VIP approves some changes that will include disabling maintenance mode. [09:02:42] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10RhinosF1) p:05Unbreak!→03Triage @Varnent: Was this tracked anywhere? I can't remember seeing anything on wikitech-l and deployments doesn't show anything nor a phab task.?? Surprised you'r... [09:05:10] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10Varnent) It is hosted externally and managed by a small internal team - so documentation is handled differently. These are also incredibly rare (first time in 5+ years), so generally not much fa... [09:07:59] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10RhinosF1) Ah, Probably offtopic for this ticket but communication is a good thing. You say 'handled differently' but does that mean not publicly? Or is there a better place to look and file prob... [09:08:10] 10Operations, 10wikimediafoundation.org: Wikimeidafoundation.org down - https://phabricator.wikimedia.org/T227400 (10RhinosF1) @Varnent: ^ [10:53:09] (03CR) 10Fomafix: "Ic162a6df86c0151ea644ff01fc268087dcf0cccd is an alternative to this change without adding new variant URLs by mapping the new URLs to the " [puppet] - 10https://gerrit.wikimedia.org/r/368248 (https://phabricator.wikimedia.org/T117845) (owner: 10Fomafix) [11:10:53] PROBLEM - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is CRITICAL: 134 ge 130 https://phabricator.wikimedia.org/T202307 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen [13:32:45] PROBLEM - Debian mirror in sync with upstream on sodium is CRITICAL: /srv/mirrors/debian is over 14 hours old. https://wikitech.wikimedia.org/wiki/Mirrors [14:37:33] is there anyone around that could look into restbase2009.codfw.wmnet? someone who could attach to the console? [14:46:55] RECOVERY - Debian mirror in sync with upstream on sodium is OK: /srv/mirrors/debian is over 0 hours old. https://wikitech.wikimedia.org/wiki/Mirrors [14:52:54] (03PS1) 10Daimona Eaytoy: Fix array shape for $wgCirrusSearchExtraIndexes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521038 (https://phabricator.wikimedia.org/T227379) [15:10:22] urandom: yeah I'll take a look [15:12:10] "Virtual Serial Port is currently in use by another session. [15:14:16] ok I'll reset, nothing on console even after killing that session [15:14:52] !log power reset restbase2009 [15:14:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:20:03] RECOVERY - Host restbase2009 is UP: PING OK - Packet loss = 0%, RTA = 36.19 ms [15:21:33] 10Operations: restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10fgiunchedi) [15:26:41] godog: thanks! [15:28:20] urandom: np! looks like it is back [16:47:31] 10Operations, 10ops-codfw: restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10Reedy) [17:12:13] RECOVERY - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is OK: (C)130 ge (W)110 ge 104.6 https://phabricator.wikimedia.org/T202307 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen [17:15:10] (03PS1) 10BryanDavis: Alphabetize metrics [debs/prometheus-pdns-exporter] - 10https://gerrit.wikimedia.org/r/521041 [17:15:12] (03PS1) 10BryanDavis: Add dnsupdate, rd, recursion, security, and udp metrics [debs/prometheus-pdns-exporter] - 10https://gerrit.wikimedia.org/r/521042 (https://phabricator.wikimedia.org/T227411) [17:15:14] (03PS1) 10BryanDavis: Log more info when `pdns_control list` fails [debs/prometheus-pdns-exporter] - 10https://gerrit.wikimedia.org/r/521043 [17:25:55] !log decommissioning restbase1017-a -- T222960 [17:26:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:26:01] T222960: Fix restbase1017's physical rack - https://phabricator.wikimedia.org/T222960 [18:19:05] PROBLEM - puppet last run on alsafi is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. [18:46:19] RECOVERY - puppet last run on alsafi is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:13:19] !log decommissioning restbase1017-b -- T222960 [20:13:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:13:25] T222960: Fix restbase1017's physical rack - https://phabricator.wikimedia.org/T222960 [20:27:01] (03PS1) 10Alex Monk: encapi proxy: Allow all of localhost [puppet] - 10https://gerrit.wikimedia.org/r/521054 [20:27:50] (03CR) 10Alex Monk: "(found when looking at encapi testing for T171188#5124728)" [puppet] - 10https://gerrit.wikimedia.org/r/521054 (owner: 10Alex Monk) [21:35:03] 10Operations, 10Puppet, 10Cloud-Services, 10cloud-services-team (Kanban): Move the main WMCS puppetmaster into the Labs realm - https://phabricator.wikimedia.org/T171188 (10Krenair) encapi works: ` diff --git a/modules/openstack/manifests/puppet/master/encapi.pp b/modules/openstack/manifests/puppet/master/... [22:17:22] (03PS1) 10Urbanecm: Refactor two testing methods for logos into one testLogos method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 [22:17:24] (03PS1) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:17:26] (03PS1) 10Urbanecm: Test if 2x logo version is 2 times bigger than 1x logo version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521181 (https://phabricator.wikimedia.org/T211413) [22:18:14] (03CR) 10jerkins-bot: [V: 04-1] Refactor two testing methods for logos into one testLogos method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 (owner: 10Urbanecm) [22:18:31] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:18:35] (03CR) 10jerkins-bot: [V: 04-1] Test if 2x logo version is 2 times bigger than 1x logo version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521181 (https://phabricator.wikimedia.org/T211413) (owner: 10Urbanecm) [22:20:05] (03PS2) 10Urbanecm: Refactor two testing methods for logos into one testLogos method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 [22:28:13] (03PS2) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:29:04] (03PS3) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:30:00] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:33:02] (03PS1) 10Urbanecm: Remove fjwiki HD logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 [22:33:22] (03PS4) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:34:21] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:37:44] (03PS2) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:39:34] (03PS3) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:39:50] (03PS5) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:40:51] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:42:41] (03PS4) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:42:57] (03PS6) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:44:07] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:45:06] (03PS5) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:45:33] (03PS7) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:46:28] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:49:45] (03PS6) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:49:55] (03PS8) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:50:56] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:52:04] (03PS7) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:52:20] (03PS9) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:53:17] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [22:57:31] (03PS8) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [22:57:48] (03PS10) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [22:58:45] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [23:01:10] (03PS9) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [23:01:21] (03PS11) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:02:18] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [23:05:23] (03PS10) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [23:05:40] (03PS12) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:05:57] (03PS11) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add a wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [23:06:37] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [23:08:11] (03PS12) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add a wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [23:08:59] (03PS13) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:09:58] (03CR) 10jerkins-bot: [V: 04-1] Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 (owner: 10Urbanecm) [23:11:05] (03PS13) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add a wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [23:11:15] (03PS14) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:14:59] (03PS3) 10Urbanecm: Refactor two testing methods for logos into one testLogos method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 [23:15:56] (03PS14) 10Urbanecm: Remove HD logos for projects with no entry in wgLogo or add a wgLogo entry [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521183 (https://phabricator.wikimedia.org/T227418) [23:16:07] (03PS15) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:16:56] (03CR) 10Urbanecm: [C: 03+2] "noop for production" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 (owner: 10Urbanecm) [23:17:58] (03Merged) 10jenkins-bot: Refactor two testing methods for logos into one testLogos method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 (owner: 10Urbanecm) [23:18:13] (03CR) 10jenkins-bot: Refactor two testing methods for logos into one testLogos method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521179 (owner: 10Urbanecm) [23:20:16] (03PS16) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:20:40] (03PS17) 10Urbanecm: Test if 1x logo exists for all HD logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521180 [23:22:12] (03PS1) 10Urbanecm: Update urbanecm's .gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/521185 [23:27:24] (03PS2) 10Urbanecm: Test if 2x logo version is 2 times bigger than 1x logo version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521181 (https://phabricator.wikimedia.org/T211413) [23:28:11] (03CR) 10jerkins-bot: [V: 04-1] Test if 2x logo version is 2 times bigger than 1x logo version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/521181 (https://phabricator.wikimedia.org/T211413) (owner: 10Urbanecm)