[00:03:05] 10Operations, 10Developer-Relations, 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3879559 (10Qgil) OK, thank you. I created the new instance: https://tools.wmflabs.org/openstack-browser/server/discourse-mediawiki.discours... [00:30:49] (03PS2) 10Dzahn: add research.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/401603 (https://phabricator.wikimedia.org/T183916) [00:31:05] 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Requesting access to Production SSH, statistics-privatedata-users, analytics-privatedata-users, perf-team for imarlier - https://phabricator.wikimedia.org/T184190#3879589 (10VColeman) a:03VColeman approved [00:32:16] (03CR) 10Dzahn: [C: 032] add research.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/401603 (https://phabricator.wikimedia.org/T183916) (owner: 10Dzahn) [00:37:32] 10Operations, 10Domains, 10Research, 10Traffic, 10Patch-For-Review: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3879615 (10Dzahn) 05Open>03Resolved a:03Dzahn https://research.wikimedia.org/ [00:38:45] is there some kind of blacklist for what headers one can store with a Swift object? [00:39:25] I'm trying to set a custom header, the PUT request succeeds but the custom header is not stored [00:51:11] (03PS1) 10Dzahn: research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) [00:51:40] (03CR) 10jerkins-bot: [V: 04-1] research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) (owner: 10Dzahn) [00:52:06] (03PS2) 10Dzahn: research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) [00:52:20] 10Operations, 10Commons, 10Multimedia, 10Traffic, and 3 others: Disable serving unpatrolled new files to Wikipedia Zero users - https://phabricator.wikimedia.org/T167400#3331813 (10Tgr) Apparently Swift has a whitelist for allowed headers and the configuration for that is [[https://bugs.launchpad.net/swift... [00:52:29] (03CR) 10jerkins-bot: [V: 04-1] research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) (owner: 10Dzahn) [00:53:25] (03PS3) 10Dzahn: research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) [00:54:19] (03CR) 10Dzahn: [C: 032] research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) (owner: 10Dzahn) [00:54:31] (03PS4) 10Dzahn: research: temp disable landing page [puppet] - 10https://gerrit.wikimedia.org/r/402456 (https://phabricator.wikimedia.org/T179871) [01:01:18] (03PS1) 10Dzahn: Revert "add research.wikimedia.org" [dns] - 10https://gerrit.wikimedia.org/r/402457 [01:02:23] (03CR) 10Bmansurov: [C: 031] Revert "add research.wikimedia.org" [dns] - 10https://gerrit.wikimedia.org/r/402457 (owner: 10Dzahn) [01:02:26] (03CR) 10Dzahn: [C: 032] Revert "add research.wikimedia.org" [dns] - 10https://gerrit.wikimedia.org/r/402457 (owner: 10Dzahn) [01:03:47] 10Operations, 10Domains, 10Research, 10Traffic, 10Patch-For-Review: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3879682 (10bmansurov) [01:04:48] 10Operations, 10Domains, 10Research, 10Traffic, 10Patch-For-Review: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3866972 (10bmansurov) 05Resolved>03Open Sorry, this is blocked on security review: T183973. [01:13:02] (03PS1) 10Madhuvishy: wikireplica: Remove dns for deleted wikis als.wiki* and mo.wik* [puppet] - 10https://gerrit.wikimedia.org/r/402458 (https://phabricator.wikimedia.org/T181925) [01:16:25] (03CR) 10Madhuvishy: [C: 032] wikireplica: Remove dns for deleted wikis als.wiki* and mo.wik* [puppet] - 10https://gerrit.wikimedia.org/r/402458 (https://phabricator.wikimedia.org/T181925) (owner: 10Madhuvishy) [01:32:03] (03PS1) 10Dzahn: research: re-enable Apache, disable cloning [puppet] - 10https://gerrit.wikimedia.org/r/402462 (https://phabricator.wikimedia.org/T179871) [01:32:35] (03CR) 10jerkins-bot: [V: 04-1] research: re-enable Apache, disable cloning [puppet] - 10https://gerrit.wikimedia.org/r/402462 (https://phabricator.wikimedia.org/T179871) (owner: 10Dzahn) [01:32:37] (03PS2) 10Dzahn: research: re-enable Apache, disable cloning [puppet] - 10https://gerrit.wikimedia.org/r/402462 (https://phabricator.wikimedia.org/T179871) [01:33:27] (03CR) 10Dzahn: [C: 032] research: re-enable Apache, disable cloning [puppet] - 10https://gerrit.wikimedia.org/r/402462 (https://phabricator.wikimedia.org/T179871) (owner: 10Dzahn) [01:38:28] (03PS1) 10Dzahn: research: re-enable git cloning of page content [puppet] - 10https://gerrit.wikimedia.org/r/402463 [02:01:11] (03PS2) 10Dzahn: research: re-enable git cloning of page content [puppet] - 10https://gerrit.wikimedia.org/r/402463 [02:01:30] (03CR) 10Dzahn: [C: 032] research: re-enable git cloning of page content [puppet] - 10https://gerrit.wikimedia.org/r/402463 (owner: 10Dzahn) [02:03:58] (03PS1) 10Dzahn: Revert "Revert "add research.wikimedia.org"" [dns] - 10https://gerrit.wikimedia.org/r/402465 [04:03:49] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 207.24 seconds [04:33:00] (03PS1) 10Gergő Tisza: Whitelist X-MediaWiki-Patrol-Status header in Swift [puppet] - 10https://gerrit.wikimedia.org/r/402471 (https://phabricator.wikimedia.org/T167400) [05:28:20] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10fullscreenorgId=1 [05:39:29] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10fullscreenorgId=1 [05:59:46] (03CR) 10Legoktm: [WIP] php7 manifests for mediawiki on stretch (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/394977 (owner: 10ArielGlenn) [06:14:35] 10Operations, 10Developer-Relations, 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3879939 (10Samwilson) I think you can change to use overlay FS by adding this in `/etc/docker/daemon.json`: ```lang=json { "storage-dri... [06:21:47] 10Operations: rebuild php-wikidiff2 and php-luasandbox for php7 and stretch - https://phabricator.wikimedia.org/T184270#3879940 (10Legoktm) I prepped a stretch backport of wikidiff2 and put it on mentors https://mentors.debian.net/package/wikidiff2 - I belive it'll need to go through backports-NEW though. I thi... [06:35:50] PROBLEM - Check size of conntrack table on mw1335 is CRITICAL: CRITICAL: nf_conntrack is 90 % full [06:37:50] RECOVERY - Check size of conntrack table on mw1335 is OK: OK: nf_conntrack is 76 % full [06:38:06] (03PS4) 10Jayprakash12345: Add Translation: namespace on Punjabi Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807) [06:40:02] (03CR) 10Jayprakash12345: "@MarcoAurelio Please remove -1. As per I have answered on task" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807) (owner: 10Jayprakash12345) [06:49:37] !log set sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on mw1335 (new jobrunner, was 120) [06:49:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:53:45] (checked via cumin, all the jobrunners now have net.netfilter.nf_conntrack_tcp_timeout_time_wait=65) [06:55:03] (03PS4) 10Jayprakash12345: Enable commons import in tawikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399556 (https://phabricator.wikimedia.org/T181774) [06:59:16] !log set sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on mw[1329-1333] (new appservers, was 120) [06:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:01:42] moritzm: all the videoscalers have net.netfilter.nf_conntrack_tcp_timeout_time_wait=120, probably becuase of the reimage? Shall we set it to 65? [07:32:40] PROBLEM - Check Varnish expiry mailbox lag on cp4026 is CRITICAL: CRITICAL: expiry mailbox lag is 2010966 [07:38:16] (03PS1) 10ArielGlenn: add labs lvm setup for dumps snapshot instances without nfs filesystems [puppet] - 10https://gerrit.wikimedia.org/r/402489 [07:40:44] (03CR) 10ArielGlenn: [C: 032] add labs lvm setup for dumps snapshot instances without nfs filesystems [puppet] - 10https://gerrit.wikimedia.org/r/402489 (owner: 10ArielGlenn) [07:47:02] (03PS8) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [07:48:33] (03CR) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/394977 (owner: 10ArielGlenn) [08:02:40] PROBLEM - Check Varnish expiry mailbox lag on cp4026 is CRITICAL: CRITICAL: expiry mailbox lag is 2109727 [08:09:44] !log re-enable eventlogging mysql consumers after database maintenance - T168414 [08:09:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:09:58] T168414: Purge all old data from EventLogging master - https://phabricator.wikimedia.org/T168414 [08:13:57] why am I doing this on a saturday? The alter tables that we had to run started on Tuesday and completed only now, so the eventlogging kafka consumers that insert data in mysql have a lot to catch up [08:14:19] and we have retention policies that will kick in early next week [08:14:55] logs are looking good, will keep an eye on metrics [08:33:34] (03PS1) 10ArielGlenn: make the snapshot labs lvm stuff specific to a snapshot profile [puppet] - 10https://gerrit.wikimedia.org/r/402569 [08:35:13] welcome to the saturday workcrew, e lukey [08:36:33] (03CR) 10ArielGlenn: [C: 032] make the snapshot labs lvm stuff specific to a snapshot profile [puppet] - 10https://gerrit.wikimedia.org/r/402569 (owner: 10ArielGlenn) [08:42:09] RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning. [08:42:44] this is me --^ [08:42:55] all eventlogging processes now running [08:45:10] ok! [08:46:57] (03PS1) 10Gergő Tisza: Handle X-MediaWiki-Patrol-Status header in Varnish [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400) [08:48:07] (03CR) 10Gergő Tisza: "Tested the concept with I522cb2b5b8a97f282fa99ca342cc6879d01808d9, seemed to work fine." [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400) (owner: 10Gergő Tisza) [08:50:39] apergos: o/ [08:50:55] heh [08:51:14] 10Operations, 10Commons, 10Multimedia, 10Traffic, and 4 others: Disable serving unpatrolled new files to Wikipedia Zero users - https://phabricator.wikimedia.org/T167400#3880051 (10Tgr) The above patches take care of the basic case (just the original file, assuming core patrolling feature) on MediaWiki, Sw... [08:53:02] ugh, dealing with labs exceptions just got too complicated for first thing sat am [08:53:15] going back to bed which is what I should have done in the first place [08:53:24] will try again when I wake up later :-P [08:55:24] ah dammit, church bells. jan 6 of course [08:55:27] gaahhhhh [08:55:34] pillow over head time. gone! [09:17:20] 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-netbox, looks like it thinks its a prod box - https://phabricator.wikimedia.org/T184242#3880063 (10hashar) [09:17:23] 10Operations, 10netops, 10Patch-For-Review: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144#3880062 (10hashar) [09:19:19] 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#3880066 (10hashar) [09:19:21] 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-redis0[12] due to systemd on trusty - https://phabricator.wikimedia.org/T184243#3880068 (10hashar) [09:19:47] 10Operations, 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#3722645 (10hashar) [09:20:11] 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-redis0[12] due to systemd on trusty - https://phabricator.wikimedia.org/T184243#3877182 (10hashar) Will be fixed by upgrading them to stretch T179371 [09:20:29] 10Operations, 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#3722645 (10hashar) [09:20:32] 10Puppet, 10Beta-Cluster-Infrastructure, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#3880073 (10hashar) [09:35:08] 10Operations, 10Puppet, 10Beta-Cluster-Infrastructure, 10media-storage: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236#3880084 (10hashar) That sounds like a puppet 4 issue. The mystery is why is Swift on production not affected by th... [10:02:40] RECOVERY - Check Varnish expiry mailbox lag on cp4026 is OK: OK: expiry mailbox lag is 7 [10:28:57] 10Operations, 10MediaWiki-extensions-Score: crackling at start of OGG renditions of MIDI files (fixed in TiMidity++ 2.14.0) - https://phabricator.wikimedia.org/T50029#3880094 (10divadsn) 05Open>03declined Migration to Fluidsynth is in progress, see T181897 :) Also, Timidity got orphaned yesterday in Debia... [10:53:39] (03PS1) 10ArielGlenn: restructure the snapshot dumps data directory setup [puppet] - 10https://gerrit.wikimedia.org/r/402574 [11:00:45] (03PS2) 10ArielGlenn: restructure the snapshot dumps data directory setup [puppet] - 10https://gerrit.wikimedia.org/r/402574 [11:09:36] (03CR) 10ArielGlenn: [C: 032] restructure the snapshot dumps data directory setup [puppet] - 10https://gerrit.wikimedia.org/r/402574 (owner: 10ArielGlenn) [11:20:49] 10Operations, 10Developer-Relations, 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3880107 (10Qgil) * I started fresh with [[ https://tools.wmflabs.org/openstack-browser/server/discourse-mw.discourse.eqiad.wmflabs | a new i... [11:34:32] (03PS1) 10ArielGlenn: snapshot testbed role for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/402576 [11:42:21] (03CR) 10ArielGlenn: [C: 032] snapshot testbed role for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/402576 (owner: 10ArielGlenn) [12:02:11] (03PS9) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [12:33:43] (03PS3) 10MarcoAurelio: Add gwtoolset to GlobalGroupPermissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/395806 [12:51:31] 10Operations: Update people.wikimedia.org with the 2017 Wikimania hackathon group photo - https://phabricator.wikimedia.org/T184338#3880371 (10Framawiki) p:05Triage>03Low [13:03:35] (03PS1) 10Gilles: Upgrade to 1.9 [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/402582 (https://phabricator.wikimedia.org/T169144) [13:06:53] (03PS1) 10Framawiki: Update group photo on people.wm.org [puppet] - 10https://gerrit.wikimedia.org/r/402583 (https://phabricator.wikimedia.org/T184338) [13:07:18] (03CR) 10jerkins-bot: [V: 04-1] Update group photo on people.wm.org [puppet] - 10https://gerrit.wikimedia.org/r/402583 (https://phabricator.wikimedia.org/T184338) (owner: 10Framawiki) [13:11:19] (03CR) 10Gilles: Handle X-MediaWiki-Patrol-Status header in Varnish (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400) (owner: 10Gergő Tisza) [13:13:40] (03PS2) 10Framawiki: Update group photo on people.wm.org [puppet] - 10https://gerrit.wikimedia.org/r/402583 (https://phabricator.wikimedia.org/T184338) [15:46:17] (03PS10) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [16:23:49] 10Operations, 10Analytics, 10Analytics-Cluster, 10Research-management: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#3880545 (10Shilad) Hi all, I’m in a position to put a GPU to use now and am happy to help if I can. I want to make sure I understand where things stand: is the is... [16:35:38] (03PS11) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [16:36:05] (03CR) 10jerkins-bot: [V: 04-1] [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 (owner: 10ArielGlenn) [16:38:21] (03PS12) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [17:01:40] 10Operations, 10Cloud-VPS, 10DNS, 10Traffic, 10Beta-Cluster-reproducible: Create some mechanism for instances in projects to modify the project Designate records - https://phabricator.wikimedia.org/T184245#3880555 (10Krenair) Might also be worth looking into TSIG, dunno if what we run (pdns IIRC?) suppor... [17:01:52] (03PS13) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [17:10:14] 10Operations, 10Developer-Relations, 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3880571 (10bd808) >>! In T180854#3880107, @Qgil wrote: > I wonder whether this is a problem of `/etc/docker/daemon.json` file/directory perm... [18:02:29] PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3fullscreenorgId=1var-site=Allvar-cache_type=miscvar-status_type=5 [18:03:18] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3fullscreenorgId=1var-site=esamsvar-cache_type=Allvar-status_type=5 [18:12:18] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3fullscreenorgId=1var-site=esamsvar-cache_type=Allvar-status_type=5 [18:12:29] RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3fullscreenorgId=1var-site=Allvar-cache_type=miscvar-status_type=5 [18:54:24] 10Operations, 10Packaging: Package Poolcounter for Debian Stretch - https://phabricator.wikimedia.org/T183385#3880675 (10bd808) [18:55:44] 10Operations, 10Packaging: rebuild php-wikidiff2 and php-luasandbox for php7 and stretch - https://phabricator.wikimedia.org/T184270#3880676 (10bd808) [19:37:26] (03PS1) 10Jcrespo: mariadb: Remove comments about partitioning on db2039 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402599 (https://phabricator.wikimedia.org/T184090) [20:56:09] PROBLEM - Long running screen/tmux on bast4002 is CRITICAL: CRIT: Long running SCREEN process. (PID: 12412, 2725098s 1728000s). [21:06:15] (03PS14) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [21:09:22] (03PS15) 10ArielGlenn: [WIP] php7 manifests for mediawiki on stretch [puppet] - 10https://gerrit.wikimedia.org/r/394977 [21:39:06] (03CR) 10Hashar: [C: 031] "Thank you Alex!" [puppet] - 10https://gerrit.wikimedia.org/r/402432 (https://phabricator.wikimedia.org/T184239) (owner: 10Paladox) [21:52:53] 10Operations, 10Developer-Relations, 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3880773 (10Qgil) a:03Qgil Thank you @bd808! I would have never found this information on my own (I had searched in Wikitech). Alright, a... [21:55:24] 10Operations, 10Developer-Relations, 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3880785 (10Qgil) [21:56:10] 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3771512 (10Qgil) [23:22:44] 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3880870 (10bd808) >>! In T180854#3880773, @Qgil wrote: > * In order to complete the installation I had to sort out the SMTP s...