[00:00:45] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add LDAP to aezell for read/write access of Grafana - https://phabricator.wikimedia.org/T211945 (10Dzahn) 05Open→03Resolved a:03Dzahn @aezell done, you have been added to the wmf group. It should work now to login. [01:16:37] PROBLEM - puppet last run on mw1301 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:47:47] RECOVERY - puppet last run on mw1301 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [01:56:11] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [02:02:11] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [02:05:57] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [02:07:39] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [02:08:45] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [02:09:31] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [03:35:05] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 924.26 seconds [04:06:24] (03PS2) 10Tulsi Bhagat: Enable 'pagemover' user group at ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) [04:06:39] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 261.40 seconds [04:17:23] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [04:19:49] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [04:20:01] (03CR) 10Mathew.onipe: wdqs: reduce hiera configs via profile defaults (039 comments) [puppet] - 10https://gerrit.wikimedia.org/r/479376 (https://phabricator.wikimedia.org/T210431) (owner: 10Mathew.onipe) [04:20:21] (03PS2) 10Mathew.onipe: wdqs: reduce hiera configs via profile defaults [puppet] - 10https://gerrit.wikimedia.org/r/479376 (https://phabricator.wikimedia.org/T210431) [04:34:31] (03CR) 10Mathew.onipe: [C: 03+1] spicerack: configure APT component/spicerack [puppet] - 10https://gerrit.wikimedia.org/r/479555 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [05:00:43] PROBLEM - HTTP availability for Varnish at eqiad on icinga1001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3fullscreenrefresh=1morgId=1 [05:01:07] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [05:02:19] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [05:03:07] RECOVERY - HTTP availability for Varnish at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3fullscreenrefresh=1morgId=1 [06:22:15] PROBLEM - Disk space on an-coord1001 is CRITICAL: DISK CRITICAL - free space: / 1765 MB (3% inode=91%) [06:25:55] RECOVERY - Disk space on an-coord1001 is OK: DISK OK [06:29:39] PROBLEM - puppet last run on mc1035 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/apt2xml] [07:00:53] RECOVERY - puppet last run on mc1035 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:02:47] Hi, I've forgot my Phabricator's 2FA, how can I reset or remove it? [09:07:47] 10Operations, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review, 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [09:08:19] 10Operations, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review, 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [09:19:03] (03CR) 10Giuseppe Lavagetto: [C: 03+1] puppet: add PuppetMaster class [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [09:21:34] (03CR) 10Giuseppe Lavagetto: [C: 03+1] puppet: add additional methods to PuppetHosts (033 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [09:21:39] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [09:22:20] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Add ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/478030 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [09:22:52] !log mask + reset-failed kafkatee default instance on weblog1001 [09:22:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:23:01] RECOVERY - Check systemd state on weblog1001 is OK: OK - running: The system is fully operational [09:25:19] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [09:52:14] (03CR) 10Mathew.onipe: puppet: add additional methods to PuppetHosts (033 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [10:01:43] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [10:02:31] PROBLEM - HTTP availability for Varnish at eqiad on icinga1001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3fullscreenrefresh=1morgId=1 [10:04:55] RECOVERY - HTTP availability for Varnish at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3fullscreenrefresh=1morgId=1 [10:05:21] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [10:39:41] (03CR) 10Mathew.onipe: [C: 03+1] Add ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/478030 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [10:52:25] (03CR) 10Mathew.onipe: puppet: add PuppetMaster class (032 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:22:05] (03PS6) 10Volans: puppet: add PuppetMaster class [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) [11:22:07] (03PS6) 10Volans: Add ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/478030 (https://phabricator.wikimedia.org/T205884) [11:22:09] (03PS5) 10Volans: icinga: fix typo in test docstring [software/spicerack] - 10https://gerrit.wikimedia.org/r/478931 (https://phabricator.wikimedia.org/T205884) [11:22:11] (03PS3) 10Volans: puppet: add additional methods to PuppetHosts [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) [11:22:17] RECOVERY - Memory correctable errors -EDAC- on mw1239 is OK: (C)4 ge (W)2 ge 1 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1var-server=mw1239var-datasource=eqiad%2520prometheus%252Fops [11:22:19] (03CR) 10Volans: "Done" (032 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:24:01] (03CR) 10Volans: "Replies inline" (034 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:30:42] (03CR) 10Mathew.onipe: [C: 03+1] puppet: add PuppetMaster class [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:35:26] (03CR) 10Mathew.onipe: puppet: add additional methods to PuppetHosts (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:39:24] (03CR) 10Volans: [C: 03+2] puppet: add PuppetMaster class [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:40:47] (03Merged) 10jenkins-bot: puppet: add PuppetMaster class [software/spicerack] - 10https://gerrit.wikimedia.org/r/477707 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:41:54] (03CR) 10Volans: [C: 03+2] Add ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/478030 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:43:20] (03Merged) 10jenkins-bot: Add ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/478030 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:44:57] (03CR) 10Volans: [C: 03+2] icinga: fix typo in test docstring [software/spicerack] - 10https://gerrit.wikimedia.org/r/478931 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:46:23] (03Merged) 10jenkins-bot: icinga: fix typo in test docstring [software/spicerack] - 10https://gerrit.wikimedia.org/r/478931 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:49:56] (03CR) 10Mathew.onipe: [C: 03+1] puppet: add additional methods to PuppetHosts (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:57:27] (03CR) 10Volans: [C: 03+2] puppet: add additional methods to PuppetHosts (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:58:05] (03PS3) 10Volans: spicerack: configure APT component/spicerack [puppet] - 10https://gerrit.wikimedia.org/r/479555 (https://phabricator.wikimedia.org/T205884) [11:59:08] (03Merged) 10jenkins-bot: puppet: add additional methods to PuppetHosts [software/spicerack] - 10https://gerrit.wikimedia.org/r/479431 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [11:59:57] (03CR) 10Volans: [C: 03+2] spicerack: configure APT component/spicerack [puppet] - 10https://gerrit.wikimedia.org/r/479555 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [12:00:55] (03CR) 10Mathew.onipe: [C: 03+1] "All Looks fine." [cookbooks] - 10https://gerrit.wikimedia.org/r/479463 (https://phabricator.wikimedia.org/T205884) (owner: 10Volans) [12:28:06] hi volans|off - could you check https://gerrit.wikimedia.org/r/#/c/operations/cookbooks/+/460731/ when you've got a minute? Thanks :) [12:32:15] Hauskatze: Hi, that somehow slipped from my inbox, but I agree with the comments left by Daniel. [12:32:50] I'll probably fix them myself later at this point [12:32:53] volans|off: ack - the text is the standard GPL license text [12:33:12] so nothing really changes except the indentation and https [12:33:25] (we ain't suposed to modify that text) [12:36:27] PROBLEM - pdfrender on scb1004 is CRITICAL: HTTP CRITICAL - No data received from host [12:37:03] Hauskatze: what do you mean? the <> should be replaced AFAIK... [12:37:39] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.003 second response time [12:37:56] volans|off: I don't think so but I'd ask legoktm :) [13:03:39] !log restarted pdfrender on scb1004 [13:03:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:19:57] (03CR) 10Zoranzoki21: "recheck" (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [14:45:19] (03CR) 10MarcoAurelio: Enable 'pagemover' user group at ur.wiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479723 (https://phabricator.wikimedia.org/T211978) (owner: 10Tulsi Bhagat) [14:55:39] (03PS1) 10BBlack: Eliminate {{zonename}} templating in favor of @Z [dns] - 10https://gerrit.wikimedia.org/r/479889 [14:55:41] (03PS1) 10BBlack: Switch to %include for geolang templating [dns] - 10https://gerrit.wikimedia.org/r/479890 [14:55:43] (03PS1) 10BBlack: Remove trailing serial comments [dns] - 10https://gerrit.wikimedia.org/r/479891 [14:55:45] (03PS1) 10BBlack: New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 [14:55:56] (03CR) 10jerkins-bot: [V: 04-1] New zone generator gen-zones.py [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [14:56:48] (03CR) 10BBlack: "^ CI failure expected until image is updated for new prereqs" [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [15:52:29] (03CR) 10Volans: "Some comment inline" (035 comments) [dns] - 10https://gerrit.wikimedia.org/r/479892 (owner: 10BBlack) [16:36:49] 10Operations, 10Operations-Software-Development, 10Patch-For-Review: Develop and deploy at least three Netbox reports to assist with data correctness and consistency - https://phabricator.wikimedia.org/T205899 (10Volans) >>! In T205899#4824154, @crusnov wrote: > Alright, going through puppetboard seems reaso... [16:38:14] (03CR) 10Volans: "LGTM, nit inline" (031 comment) [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/479583 (https://phabricator.wikimedia.org/T205899) (owner: 10CRusnov) [16:39:15] 10Operations, 10Traffic, 10netops, 10Patch-For-Review, 10Performance-Team (Radar): Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006 (10Krinkle) [16:41:30] (03Abandoned) 10Ladsgroup: ores: Change result serializer to json [puppet] - 10https://gerrit.wikimedia.org/r/477302 (https://phabricator.wikimedia.org/T206333) (owner: 10Ladsgroup) [17:48:51] (03CR) 10MarcoAurelio: [C: 03+1] Use standard version of plain-text GPL (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/460731 (owner: 10Legoktm) [19:41:59] PROBLEM - graphite-labs.wikimedia.org on labmon1001 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 398 bytes in 0.002 second response time [19:47:32] (03PS1) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [19:47:56] 10Operations, 10Phabricator, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10mmodell) I created an #acl_blog-admins project and set up the blog edit form so that members should be able to edit the policy controls. @ema: can you see the policy... [19:48:07] RECOVERY - graphite-labs.wikimedia.org on labmon1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1547 bytes in 0.012 second response time [19:48:09] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:51:06] (03PS2) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [19:51:54] (03PS3) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [19:51:59] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:52:01] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:52:49] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:53:30] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:55:36] (03PS4) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [19:56:30] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:57:04] (03PS5) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [19:57:44] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:57:56] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:58:48] (03PS6) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [19:58:53] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [19:59:42] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [20:00:16] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [20:00:18] (03PS7) 10Paladox: phabricator: Fix loading of php-extensions [puppet] - 10https://gerrit.wikimedia.org/r/479909 [20:00:29] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/479909 (owner: 10Paladox) [20:01:55] PROBLEM - Nginx local proxy to apache on mw1286 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.010 second response time [20:02:35] PROBLEM - Apache HTTP on mw1286 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.002 second response time [20:03:09] RECOVERY - Nginx local proxy to apache on mw1286 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.040 second response time [20:03:49] RECOVERY - Apache HTTP on mw1286 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 616 bytes in 0.030 second response time [20:06:20] (03CR) 1020after4: [C: 03+2] Scap prep should use latest MediaWiki version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/474971 (owner: 10Thcipriani) [20:07:26] (03Merged) 10jenkins-bot: Scap prep should use latest MediaWiki version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/474971 (owner: 10Thcipriani) [20:10:43] (03CR) 10jenkins-bot: Scap prep should use latest MediaWiki version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/474971 (owner: 10Thcipriani) [20:28:38] (03PS1) 10MaxSem: Undeploy Listings from dewikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479911 (https://phabricator.wikimedia.org/T206102) [22:11:45] PROBLEM - Nginx local proxy to apache on mw1281 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.011 second response time [22:12:13] PROBLEM - HHVM rendering on mw1281 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1309 bytes in 0.006 second response time [22:12:31] PROBLEM - Apache HTTP on mw1281 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.002 second response time [22:12:57] RECOVERY - Nginx local proxy to apache on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.054 second response time [22:13:27] RECOVERY - HHVM rendering on mw1281 is OK: HTTP OK: HTTP/1.1 200 OK - 75637 bytes in 0.787 second response time [22:13:43] RECOVERY - Apache HTTP on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 616 bytes in 0.034 second response time [23:47:39] (03PS2) 10Urbanecm: Upload custom minerva logo for cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479379 (https://phabricator.wikimedia.org/T210979) [23:48:14] (03PS1) 10Urbanecm: Use new minerva logos for cswiki in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479963 (https://phabricator.wikimedia.org/T210979) [23:49:08] (03PS3) 10Urbanecm: Upload custom minerva logo for cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479379 (https://phabricator.wikimedia.org/T210979) [23:49:18] (03PS2) 10Urbanecm: Use new minerva logos for cswiki in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/479963 (https://phabricator.wikimedia.org/T210979)