[00:03:42] !log split member-range ge-3/0/0 to ge-3/0/38 on asw-b-codfw [00:03:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:07:06] (03CR) 10Dzahn: DNS: Add production DNS enties for cloudcontrol2001-dev and cloudvirt200[123]-dev (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/486504 (https://phabricator.wikimedia.org/T214448) (owner: 10Papaul) [00:08:01] 10Operations: wmf-auto-reimage-host: icinga downtime error - https://phabricator.wikimedia.org/T214314 (10Volans) The problem was misdiagnosed here, we have two issues, one is related to Icinga and is anyway considered a warning and doesn't make the script stop, the other is the real error instead. ` 16:36:10 |... [00:16:48] elukey: do you know where the nutcracker yaml config for wikitech is set in puppet? [00:17:58] (03CR) 10CRusnov: [C: 03+1] "LGTM" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/486530 (owner: 10Volans) [00:26:35] 10Operations, 10ops-eqiad, 10ops-eqsin, 10netops: Deploy cr2-eqsin - https://phabricator.wikimedia.org/T213121 (10ayounsi) a:05ayounsi→03Cmjohnson [00:30:48] AaronSchulz: modules/profile/manifests/mediawiki/nutcracker.pp may be what you are looking for [00:31:12] oh, wikitech. Hmmmm [00:36:09] AaronSchulz: wikitech's looks to be generated by profile::openstack::base::nutcracker [00:45:24] ah, thanks [01:02:21] RECOVERY - MariaDB Slave Lag: s8 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 138.03 seconds [01:07:32] 10Operations: wmf-auto-reimage-host: icinga downtime error - https://phabricator.wikimedia.org/T214314 (10Volans) Forgot to mention that as a quick fix passing the option `--rename-mgmt`` explicitely should avoid the problem. [02:11:57] (03PS2) 10Papaul: DNS: Add production DNS enties for cloudcontrol2001-dev and cloudvirt200[123]-dev [dns] - 10https://gerrit.wikimedia.org/r/486504 (https://phabricator.wikimedia.org/T214448) [02:17:49] (03CR) 10Dzahn: [V: 03+1 C: 03+1] DNS: Add production DNS enties for cloudcontrol2001-dev and cloudvirt200[123]-dev [dns] - 10https://gerrit.wikimedia.org/r/486504 (https://phabricator.wikimedia.org/T214448) (owner: 10Papaul) [03:02:16] 10Operations, 10ops-eqiad, 10DBA: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10Marostegui) 05Stalled→03Resolved I am going to close this for now, as it has been 10 days without issues: ` root@db1115:~# free -g tota... [03:06:35] 10Operations, 10ops-eqiad, 10DBA, 10Patch-For-Review: db1114 crashed - https://phabricator.wikimedia.org/T214720 (10Marostegui) a:03Cmjohnson @jcrespo +1 to reimage/reclone from an existing host (or mariabackup!) @Cmjohnson I guess we need to open a case with Dell about this host and upgrade BIOS/firmware? [03:08:00] 10Operations, 10ops-eqiad, 10Analytics: Rack A2's hosts alarm for PSU broken - https://phabricator.wikimedia.org/T212861 (10Marostegui) [03:08:07] 10Operations, 10ops-eqiad, 10Analytics, 10DBA, 10Patch-For-Review: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10Marostegui) 05Open→03Resolved I believe there is nothing else pending here, and this was re-opened just to get an answer from Chris, which was done. G... [03:15:15] (03PS1) 10Paladox: WIP: Update gerrit to 2.16.4 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/486711 [03:15:31] (03Abandoned) 10Paladox: WIP: Update gerrit to 2.16 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/463509 (owner: 10Paladox) [03:15:51] (03PS2) 10Paladox: WIP: Update gerrit to 2.16.4 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/486711 [03:23:54] !log Convert all tables on incubatorwiki to innodb to fix s3 thread - T213670 [03:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:23:57] T213670: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 [03:24:04] 10Operations, 10Analytics, 10Analytics-Kanban, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) s3 thread broke with: ` Last_SQL_Error: Error 'Got error 22 "Invalid argument" from storage engine TokuDB' on query. Default database: 'incu... [03:26:29] (03PS1) 10Marostegui: dbstore.my.cnf.erb: Make InnoDB default [puppet] - 10https://gerrit.wikimedia.org/r/486712 (https://phabricator.wikimedia.org/T213670) [03:29:01] (03PS3) 10Paladox: WIP: Update gerrit to 2.16.4 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/486711 [03:32:22] (03PS4) 10Paladox: WIP: Update gerrit to 2.16.4 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/486711 [03:37:00] (03CR) 10Marostegui: "Puppet compiler looks good: https://puppet-compiler.wmflabs.org/compiler1002/14488/" [puppet] - 10https://gerrit.wikimedia.org/r/486712 (https://phabricator.wikimedia.org/T213670) (owner: 10Marostegui) [03:37:54] (03CR) 10Marostegui: "I have changed this dynamically to avoid any more creations for now." [puppet] - 10https://gerrit.wikimedia.org/r/486712 (https://phabricator.wikimedia.org/T213670) (owner: 10Marostegui) [03:56:43] PROBLEM - PyBal backends health check on lvs1006 is CRITICAL: PYBAL CRITICAL - CRITICAL - mathoid_10042: Servers kubernetes1001.eqiad.wmnet, kubernetes1004.eqiad.wmnet are marked down but pooled [03:56:45] PROBLEM - Check systemd state on wdqs1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [03:59:03] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - mathoid_10042: Servers kubernetes1001.eqiad.wmnet, kubernetes1003.eqiad.wmnet are marked down but pooled [03:59:19] RECOVERY - PyBal backends health check on lvs1006 is OK: PYBAL OK - All pools are healthy [04:00:02] 10Operations, 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) All the tables on `incubatorwiki` are now InnoDB and replication is catching up. [04:00:21] RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy [04:04:33] RECOVERY - Check systemd state on wdqs1004 is OK: OK - running: The system is fully operational [05:05:31] PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 229, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [05:06:35] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is CRITICAL: CRITICAL - failed 63 probes of 375 (alerts on 35) - https://atlas.ripe.net/measurements/1791212/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts [05:11:53] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is OK: OK - failed 23 probes of 375 (alerts on 35) - https://atlas.ripe.net/measurements/1791212/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts [05:43:07] PROBLEM - puppet last run on proton1002 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 4 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Exec[absent_ensure_members],Exec[ops_ensure_members],Exec[wikidev_ensure_members] [06:09:31] RECOVERY - puppet last run on proton1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:30:02] 10Operations, 10Analytics, 10Analytics-EventLogging, 10EventBus, and 3 others: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10Krinkle) 05Open→03Resolved a:03Krinkle [06:30:25] PROBLEM - puppet last run on ms-be1035 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] [06:32:05] PROBLEM - puppet last run on phab1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] [06:38:28] 10Operations, 10TechCom-RFC, 10Traffic, 10Wikipedia-Android-App-Backlog, and 3 others: RFC: API-driven web front-end - https://phabricator.wikimedia.org/T111588 (10Krinkle) [06:50:52] 10Operations, 10Wikimedia-Mailing-lists: Adding administrator to mailing list for Wikimedia New Zealand - https://phabricator.wikimedia.org/T214271 (10Podzemnik) https://nz.wikimedia.org/wiki/Main_Page - the project of the chapter was canceled as there were not enough interest in NZ to create a chapter We hop... [06:56:49] RECOVERY - puppet last run on ms-be1035 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:58:31] RECOVERY - puppet last run on phab1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [07:02:39] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 231, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [07:44:04] 10Operations, 10DBA, 10MediaWiki-extensions-Translate, 10Datacenter-Switchover-2018, and 2 others: DBPerformance warning "Query returned 22186 rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki - https://phabricator.wikimedia.org/T204026 (10Krinkle) [08:06:51] 10Operations, 10Analytics, 10User-Elukey: notebook/stat server(s) running out of memory - https://phabricator.wikimedia.org/T212824 (10elukey) Thanks a lot for all the work :) I am leaning towards having a cgroup that limits CPU/memory usage of all processes to something like 80/85%, so hopefully this will... [08:07:05] 10Operations, 10Analytics, 10User-Elukey: notebook/stat server(s) running out of memory - https://phabricator.wikimedia.org/T212824 (10elukey) p:05Normal→03High a:03elukey [08:30:17] PROBLEM - BGP status on cr4-ulsfo is CRITICAL: BGP CRITICAL - AS6939/IPv4: Connect, AS6939/IPv6: Active https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [08:38:18] (03PS5) 10KartikMistry: WIP: Cron to run script to purge old CX drafts [puppet] - 10https://gerrit.wikimedia.org/r/486454 (https://phabricator.wikimedia.org/T189091) [08:45:25] (03CR) 10KartikMistry: "> @KartikMistry there are plans to use a new define for mediawiki" [puppet] - 10https://gerrit.wikimedia.org/r/486454 (https://phabricator.wikimedia.org/T189091) (owner: 10KartikMistry) [09:02:39] RECOVERY - BGP status on cr4-ulsfo is OK: BGP OK - up: 99, down: 1, shutdown: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [09:23:28] (03PS7) 10Ammarpad: Add 'Author' namespace in Sanskrit Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553) [09:26:44] (03PS8) 10Ammarpad: Add 'Author' namespace in Sanskrit Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553) [09:28:57] (03CR) 10Ammarpad: Add 'Author' namespace in Sanskrit Wikisource (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553) (owner: 10Ammarpad) [09:44:37] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 178693.56 seconds [10:01:45] PROBLEM - MariaDB Slave Lag: s8 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1001.73 seconds [11:06:48] !log force rebooting icinga1001 (no ping, no ssh, stuck console) [11:06:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:17:52] (03CR) 10Zoranzoki21: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553) (owner: 10Ammarpad) [11:27:48] 10Operations, 10monitoring: icinga1001 crashed - https://phabricator.wikimedia.org/T214760 (10Volans) [11:29:08] PROBLEM - MariaDB Slave Lag: s8 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 6244.29 seconds [12:03:51] (03CR) 10Ammarpad: "> The manual rebase performed here is faulty. I suggest you abandon" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) (owner: 10Ammarpad) [13:04:00] (03PS8) 10Ammarpad: Enable blocking feature of AbuseFilter in zh.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) [13:33:35] 10Operations, 10Wikimedia-Mailing-lists: Adding administrator to mailing list for Wikimedia New Zealand - https://phabricator.wikimedia.org/T214271 (10Dzahn) @Podzemnik Thank you! It all sounds good. Just let me know which email address to use. [14:35:26] 10Operations, 10monitoring: WMF's Grafana installation does not follow Wikimedia's visual identity guidelines - https://phabricator.wikimedia.org/T214762 (10jcrespo) [14:58:10] 10Operations, 10Wikimedia-Mailing-lists: Adding administrator to mailing list for Wikimedia New Zealand - https://phabricator.wikimedia.org/T214271 (10Podzemnik) @Dzahn My e-mail would be good, it's michal.klajban@gmail.com [15:13:49] 10Operations, 10User-herron: Improve visibility of incoming operations tasks - https://phabricator.wikimedia.org/T197624 (10Aklapper) As there is no progress in T205258, I guess next step would be using the API to see if that could work around T205258. Something like ` #!/bin/bash for i in Txxxx Txxxx Txxxx do... [17:15:39] (03PS5) 10Hashar: scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 [17:17:08] (03CR) 10jerkins-bot: [V: 04-1] scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar) [17:22:28] (03CR) 10Hashar: "PS5 reworks the ThreadPool execution. The function returns the ImageFSM asynchronously and we then process the results to update all_image" (033 comments) [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar) [17:23:21] (03CR) 10Hashar: "PS5 adds scan_workers configuration setting to tweak the number of threads being used when scanning images." [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar) [18:15:30] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 661.89 seconds [18:20:24] PROBLEM - Kafka MirrorMaker main-eqiad_to_main-codfw max lag in last 10 minutes on icinga1001 is CRITICAL: 1.909e+06 gt 1e+05 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=codfw+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_main-codfw [19:15:58] (03CR) 10Zoranzoki21: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) (owner: 10Ammarpad) [19:17:03] (03CR) 10jerkins-bot: [V: 04-1] Enable blocking feature of AbuseFilter in zh.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) (owner: 10Ammarpad) [19:18:35] (03CR) 10Zoranzoki21: [C: 04-1] "BTW, you need one empty line at end of file." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) (owner: 10Ammarpad) [19:25:08] RECOVERY - Kafka MirrorMaker main-eqiad_to_main-codfw max lag in last 10 minutes on icinga1001 is OK: (C)1e+05 gt (W)1e+04 gt 0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=codfw+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_main-codfw [19:28:24] 10Operations, 10MediaWiki-General-or-Unknown, 10Multimedia, 10media-storage: Lost file Juan_Guaidó.jpg - https://phabricator.wikimedia.org/T213655 (10User100100) Seems like that this file is now reuploaded to Commons, so there is no need to restore anything. https://commons.wikimedia.org/wiki/File:Juan_Gu... [19:42:32] 10Operations, 10Continuous-Integration-Config: CI errors not being displayed in console log - https://phabricator.wikimedia.org/T214726 (10jbond) [19:55:03] 10Operations, 10Security-Team, 10Wikimedia-Site-requests, 10Patch-For-Review: Enable csp-report-only mode everywhere - https://phabricator.wikimedia.org/T207900 (10Tomybrz) Any Update? [20:06:19] (03PS9) 10Ammarpad: Enable blocking feature of AbuseFilter in zh.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) [20:07:29] (03CR) 10Ammarpad: "> BTW, you need one empty line at end of file." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) (owner: 10Ammarpad) [20:08:25] (03CR) 10D3r1ck01: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486103 (https://phabricator.wikimedia.org/T210364) (owner: 10Ammarpad) [20:22:02] (03CR) 10Framawiki: "> Patch Set 6:" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553) (owner: 10Ammarpad) [20:38:42] PROBLEM - MariaDB Slave SQL: x1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1062, Errmsg: Could not execute Write_rows_v1 event on table wikishared.echo_unread_wikis: Duplicate entry 19236935-jawiki for key echo_unread_wikis_user_wiki, Error_code: 1062: handler error HA_ERR_FOUND_DUPP_KEY: the events master log db1069-bin.000322, end_log_pos 1004044989 [20:40:57] (03CR) 10Ammarpad: Add 'Author' namespace in Sanskrit Wikisource (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553) (owner: 10Ammarpad) [20:52:50] PROBLEM - MariaDB Slave Lag: x1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1041.39 seconds [20:55:46] PROBLEM - puppet last run on mw1341 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:55:59] 10Operations, 10Traffic: Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) - https://phabricator.wikimedia.org/T112316 (10Krinkle) [20:56:03] (03CR) 10Ammarpad: [C: 03+1] Removed WikibaseQuality from extensions-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486509 (https://phabricator.wikimedia.org/T208499) (owner: 10Zoranzoki21) [21:22:04] RECOVERY - puppet last run on mw1341 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [21:24:56] 10Operations, 10User-herron: Improve visibility of incoming operations tasks - https://phabricator.wikimedia.org/T197624 (10Aklapper) Doing this now by moving public open tasks from `Backlog` to `Acknowledged` column on #operations workboard by using the Conduit API. Note that `@Phabricator_maintenance` rightf... [21:29:08] RECOVERY - MariaDB Slave Lag: s4 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 295.11 seconds [21:58:46] 10Operations, 10monitoring: WMF's Grafana installation does not follow Wikimedia's visual identity guidelines - https://phabricator.wikimedia.org/T214762 (10faidon) It's tricky, but I think the one we use one is probably the right one and this should be declined. See T212674 for context. [22:11:45] 10Operations, 10Continuous-Integration-Config: CI errors not being displayed in console log - https://phabricator.wikimedia.org/T214726 (10hashar) The job ran for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/486513/ . I downloaded it and ran the tests while disabling mutliprocessing: ` $ bundle exec... [22:55:17] (03CR) 10Gehel: [C: 03+1] icinga: add context manager for downtimed hosts [software/spicerack] - 10https://gerrit.wikimedia.org/r/486530 (owner: 10Volans) [22:58:47] 10Operations, 10monitoring: WMF's Grafana installation does not follow Wikimedia's visual identity guidelines - https://phabricator.wikimedia.org/T214762 (10Krinkle) My 2c: We may indeed want to keep using the "community logo". However, using it on a black background is imho still a problem. [23:11:48] PROBLEM - HHVM rendering on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:11:49] 10Operations, 10User-herron: Improve visibility of incoming operations tasks - https://phabricator.wikimedia.org/T197624 (10Aklapper) Done; terminated the API token of `@Phabricator_maintenance`. I stopped at `T213918` as I'm not sure how recent the tasks are acknowledged. [23:12:58] RECOVERY - HHVM rendering on mw1341 is OK: HTTP OK: HTTP/1.1 200 OK - 78100 bytes in 4.994 second response time [23:21:14] 10Operations, 10puppet-compiler: puppet: compiler-update-facts error and warning - https://phabricator.wikimedia.org/T214472 (10Peachey88) [23:22:15] 10Operations, 10Puppet, 10Discovery-Search, 10Maps: Fix maps puppet to make sure apt-get update runs after configuration change - https://phabricator.wikimedia.org/T214073 (10Peachey88) [23:58:09] (03CR) 10Krinkle: [C: 03+2] In LocalSettings.php use a relative path to CommonSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486004 (owner: 10Tim Starling) [23:59:15] (03Merged) 10jenkins-bot: In LocalSettings.php use a relative path to CommonSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486004 (owner: 10Tim Starling)