[00:19:26] 10Operations, 10Analytics-Kanban, 10Patch-For-Review: Puppet admin module should support adding system users to managed groups - https://phabricator.wikimedia.org/T174465#3914060 (10mpopov) >>! In T174465#3908880, @MoritzMuehlenhoff wrote: > This is solely for T174110 or are we anticipating other use cases?... [00:24:54] 10Operations, 10Commons, 10TimedMediaHandler-Transcode, 10Wikimedia-Video, and 3 others: Commons video transcoders have over 6500 tasks in the backlog. - https://phabricator.wikimedia.org/T153488#3914068 (10brion) [00:24:58] 10Operations, 10TimedMediaHandler: Assign 3 more servers to video scaler duty - https://phabricator.wikimedia.org/T114337#3914066 (10brion) 05Open>03stalled Abandoning this for now; will reopen if necessary once closer to VP9. (We're pretty close though with the Stretch update.) [00:25:05] (03PS2) 10Chico Venancio: Add Chicocvenancio's key for Cloud Services [labs/private] - 10https://gerrit.wikimedia.org/r/405376 (https://phabricator.wikimedia.org/T185273) [00:56:34] 10Operations, 10Puppet: os_version strict distro check doesn't work - https://phabricator.wikimedia.org/T185345#3913626 (10faidon) I can't reproduce. I'm testing with this for example: ``` $is_trusty = os_version('ubuntu trusty') notice("is trusty is ${is_trusty}") $is_stretch = os_version('debian stretch') n... [00:57:02] 10Operations, 10Analytics-Data-Quality, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3914151 (10Tbayer) >>! In T185350#3913845, @Nuria wrote: > Can we have an example that is not a media type? A small example that came up some w... [00:59:00] 10Operations, 10Ops-Access-Requests: Requesting access to bast1001, stat1005, stat1006 for risler - https://phabricator.wikimedia.org/T185356#3914155 (10RobH) p:05Triage>03Normal [01:00:54] 10Operations, 10Ops-Access-Requests: Requesting access to bast1001, stat1005, stat1006 for risler - https://phabricator.wikimedia.org/T185356#3913933 (10RobH) @Ramsey-WMF: There are multiple access groups that can login to those machines, please review https://wikitech.wikimedia.org/wiki/Analytics/Data_access#... [01:13:42] (03PS1) 10Faidon Liambotis: wmflib: make os_version spec more extensive [puppet] - 10https://gerrit.wikimedia.org/r/405387 (https://phabricator.wikimedia.org/T185345) [01:15:40] (03CR) 10Faidon Liambotis: [C: 032] wmflib: make os_version spec more extensive [puppet] - 10https://gerrit.wikimedia.org/r/405387 (https://phabricator.wikimedia.org/T185345) (owner: 10Faidon Liambotis) [01:19:23] 10Operations, 10Puppet, 10Patch-For-Review: os_version strict distro check doesn't work - https://phabricator.wikimedia.org/T185345#3914225 (10faidon) I just pushed a change to make the rspec more extensive, including a test case for the scenario that you described here. It seems to pass fine, so I merged th... [01:30:01] 10Operations, 10Domains, 10Traffic, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3914229 (10Dzahn) Is a wiki page not an option for this content? It seems mostly text and images. Has that been considered? Assu... [01:38:02] 10Operations, 10Domains, 10Traffic, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3914235 (10Volker_E) @Dzahn Yes, a wiki page was considered as option, as well as where the page repo should live. Both things hav... [01:40:10] 10Operations, 10Domains, 10Traffic, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3911827 (10Platonides) They seem to have one: https://github.com/wikimedia/WikimediaUI-Style-Guide (maybe it is already in gerrit?... [01:47:59] 10Operations, 10Domains, 10Traffic, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3914281 (10Dzahn) @Volker_E Gotcha! So you are requesting web space to clone to from Github? Pulling down a git repo from an exte... [01:54:14] 10Operations, 10Ops-Access-Requests: Requesting access to stat1004, stat1005, stat1006 for mneisler - https://phabricator.wikimedia.org/T184838#3914289 (10chelsyx) Yes, @MNeisler will support me with data analysis. But I'm not a manager. I hope @JKatzWMF 's approval counts. [03:24:54] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 836.28 seconds [03:32:52] !log bootstrapping restbase2011-a - T184100 [03:33:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:33:05] T184100: Reprovision legacy Cassandra nodes into new cluster - https://phabricator.wikimedia.org/T184100 [03:47:50] 10Operations, 10Analytics-Data-Quality, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3914372 (10Tbayer) >>! In T185350#3914151, @Tbayer wrote: ... > 253193 bytes matches the gzipped size of [[https://en.wikipedia.org/api/rest_v1... [03:58:04] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 247.74 seconds [03:59:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [04:19:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [05:12:24] PROBLEM - HHVM rendering on mw1294 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:14] RECOVERY - HHVM rendering on mw1294 is OK: HTTP OK: HTTP/1.1 200 OK - 74266 bytes in 0.154 second response time [05:29:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [05:39:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [06:16:14] PROBLEM - Router interfaces on cr2-codfw is CRITICAL: CRITICAL: host 208.80.153.193, interfaces up: 120, down: 1, dormant: 0, excluded: 0, unused: 0 [06:16:35] PROBLEM - Router interfaces on cr1-eqord is CRITICAL: CRITICAL: host 208.80.154.198, interfaces up: 37, down: 1, dormant: 0, excluded: 0, unused: 0 [06:29:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [06:49:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [06:55:31] 10Operations, 10Analytics-Data-Quality, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3914447 (10Nuria) Both 18550 and 15722 are divisors of 253193 (13 and 16) . Requests with both sizes (18550 and 15722) come from the identic... [07:54:55] PROBLEM - Varnish HTTP text-backend - port 3128 on cp3030 is CRITICAL: HTTP CRITICAL - No data received from host [07:57:45] PROBLEM - Check Varnish expiry mailbox lag on cp3030 is CRITICAL: CRITICAL: expiry mailbox lag is 2225658 [07:58:04] PROBLEM - Varnish HTTP text-backend - port 3128 on cp3032 is CRITICAL: HTTP CRITICAL - No data received from host [07:59:04] RECOVERY - Varnish HTTP text-backend - port 3128 on cp3030 is OK: HTTP OK: HTTP/1.1 200 OK - 220 bytes in 0.168 second response time [08:03:04] RECOVERY - Varnish HTTP text-backend - port 3128 on cp3032 is OK: HTTP OK: HTTP/1.1 200 OK - 220 bytes in 0.168 second response time [08:04:44] RECOVERY - Router interfaces on cr1-eqord is OK: OK: host 208.80.154.198, interfaces up: 39, down: 0, dormant: 0, excluded: 0, unused: 0 [08:05:35] RECOVERY - Router interfaces on cr2-codfw is OK: OK: host 208.80.153.193, interfaces up: 122, down: 0, dormant: 0, excluded: 0, unused: 0 [08:07:44] RECOVERY - Check Varnish expiry mailbox lag on cp3030 is OK: OK: expiry mailbox lag is 13 [09:46:50] mmh cp3030 and cp3032 above are both text hosts [09:47:19] it looks like they were having a hard time coping with mbox lag a the time of the alerts [10:06:04] PROBLEM - Nginx local proxy to apache on mw1315 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.007 second response time [10:07:04] RECOVERY - Nginx local proxy to apache on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 0.161 second response time [10:49:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [11:19:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [11:49:55] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [12:00:44] (03CR) 10Thiemo Kreuz (WMDE): [C: 04-1] "Something here doesn't look right. See my inline comments." (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/404148 (https://phabricator.wikimedia.org/T184865) (owner: 10Biplab Anand) [12:18:44] (03Restored) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:22:37] (03PS11) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) [12:22:49] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:23:46] (03PS12) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) [12:23:51] (03PS13) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) [12:25:18] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:25:24] (03PS14) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) [12:26:49] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:30:22] (03CR) 10MarcoAurelio: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:31:50] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:32:36] (03Abandoned) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [12:49:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [12:53:50] !log bootstrapping restbase2011-b - T184100 [12:54:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:54:06] T184100: Reprovision legacy Cassandra nodes into new cluster - https://phabricator.wikimedia.org/T184100 [13:07:12] 10Puppet, 10Beta-Cluster-Infrastructure, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#3914613 (10Ladsgroup) [13:07:16] 10Puppet, 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3914611 (10Ladsgroup) 05Open>03Resolved We usually don't close tickets before reporting them un... [13:12:52] (03PS1) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) [13:14:55] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:15:29] (03CR) 10MarcoAurelio: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:16:02] (03CR) 10Faidon Liambotis: "\o/" [software/cumin] (debian) - 10https://gerrit.wikimedia.org/r/405320 (owner: 10Volans) [13:17:29] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:19:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [13:23:15] (03CR) 10MarcoAurelio: "ERROR: Error fetching remote repo 'origin' -- weird." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:23:18] (03CR) 10Faidon Liambotis: [C: 04-1] lxc: Fix support for stretch (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/405208 (https://phabricator.wikimedia.org/T180377) (owner: 10Paladox) [13:23:24] (03CR) 10MarcoAurelio: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:25:27] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:29:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [13:32:31] (03PS2) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) [13:34:34] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:35:50] (03CR) 10MarcoAurelio: "Urbanecm: I can't understand very well the jenkins failures. Can you help me understand them? Thanks." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:40:11] (03CR) 10Zoranzoki21: "Marco, can I try any?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [13:59:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [14:00:44] (03CR) 10MarcoAurelio: "> Marco, can I try any?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [14:31:56] 10Puppet, 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3914744 (10Krenair) What do you mean 'before reporting them'? [14:48:31] (03CR) 10MarcoAurelio: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [14:50:32] (03CR) 10jerkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [14:54:08] (03CR) 10MarcoAurelio: "remaining is some sort of db/docroot thing..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [14:59:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [15:00:58] 10Puppet, 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3914794 (10Ladsgroup) Scoring platform team gives out monthly reports (Like Weekly summary of Wikid... [15:09:32] (03CR) 10Zoranzoki21: "> > Marco, can I try any?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405421 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio) [15:10:37] (03PS2) 10Zoranzoki21: Add https://audiovis.nac.gov.pl to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/404088 (https://phabricator.wikimedia.org/T184853) (owner: 10Urbanecm) [15:10:45] (03CR) 10jerkins-bot: [V: 04-1] Add https://audiovis.nac.gov.pl to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/404088 (https://phabricator.wikimedia.org/T184853) (owner: 10Urbanecm) [15:11:00] (03CR) 10Zoranzoki21: [C: 031] Add https://audiovis.nac.gov.pl to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/404088 (https://phabricator.wikimedia.org/T184853) (owner: 10Urbanecm) [15:29:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [16:09:55] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [16:17:49] 10Operations, 10Puppet, 10Patch-For-Review: os_version strict distro check doesn't work - https://phabricator.wikimedia.org/T185345#3914868 (10Andrew) 05Open>03Invalid ok, what I've learned is that my test puppet runs are swallowing all warning and notices. So 100% of my tests are invalid and I don't kn... [16:22:31] (03PS1) 10Andrew Bogott: mediawiki_vagrant: for distro warning, use notify [puppet] - 10https://gerrit.wikimedia.org/r/405532 [16:22:56] (03CR) 10jerkins-bot: [V: 04-1] mediawiki_vagrant: for distro warning, use notify [puppet] - 10https://gerrit.wikimedia.org/r/405532 (owner: 10Andrew Bogott) [16:24:03] (03PS2) 10Andrew Bogott: mediawiki_vagrant: for distro warning, use notify [puppet] - 10https://gerrit.wikimedia.org/r/405532 [16:24:25] (03CR) 10jerkins-bot: [V: 04-1] mediawiki_vagrant: for distro warning, use notify [puppet] - 10https://gerrit.wikimedia.org/r/405532 (owner: 10Andrew Bogott) [16:26:25] (03CR) 10Andrew Bogott: [V: 032 C: 032] mediawiki_vagrant: for distro warning, use notify [puppet] - 10https://gerrit.wikimedia.org/r/405532 (owner: 10Andrew Bogott) [16:39:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [16:44:47] 10Puppet, 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3914913 (10Krenair) So you hold your resolved tasks open for up to a month after resolution? But ye... [16:57:20] !log bootstrapping restbase2011-c - T184100 [16:57:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:57:33] T184100: Reprovision legacy Cassandra nodes into new cluster - https://phabricator.wikimedia.org/T184100 [17:36:13] !log forced bbu learn cycle on analytics1038 (cache policy flapping from WriteBack to WriteThrough) [17:36:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:38:50] 10Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-General-or-Unknown: Special:Import error: "Import failed: Could not open import file" - https://phabricator.wikimedia.org/T17000#3914981 (10valerio.bozzolan) As an additional note, it also affects it.wikiversity importing [this page](https://it.wikipedi... [17:39:55] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [17:49:25] (03CR) 10Thiemo Kreuz (WMDE): [C: 04-1] Add "Portal" namespace on it.wikiquote (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405024 (https://phabricator.wikimedia.org/T185232) (owner: 10Jayprakash12345) [17:54:35] 10Puppet, 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Puppet broken on deployment-ores01 due to missing hieradata - https://phabricator.wikimedia.org/T184478#3914991 (10Ladsgroup) At start it was weekly but things got complicated. This needs rethinking now.... [17:57:10] (03PS3) 10Jayprakash12345: Add "Portal" namespace on it.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405024 (https://phabricator.wikimedia.org/T185232) [18:09:55] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [18:19:54] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [18:36:55] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3915006 (10Isarra) >>! In T133410#3909349, @Deskana wrote: > I don't want deployment of TemplateStyles to be a moving target—as it has be... [18:49:55] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [19:26:50] Hi, Please deploy this: https://gerrit.wikimedia.org/r/#/c/404095/ [19:28:54] I can test [19:44:29] Zoranzoki21: 1) it hasn't been reviewed yet, 2) its the weekend and hasn't been scheduled for deployment yet 3) its not a UBN scenario to do a weekend deployment tbh [19:44:45] ok [19:49:55] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [20:19:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough [21:10:18] (03CR) 10Thiemo Kreuz (WMDE): "Can you please explain what kinds of "optimizations" have been done here?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402618 (https://phabricator.wikimedia.org/T177726) (owner: 10Odder) [21:49:10] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] Update path to loadExitNodes.php [puppet] - 10https://gerrit.wikimedia.org/r/399434 (owner: 10Reedy) [21:52:51] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] Add "Portal" namespace on it.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405024 (https://phabricator.wikimedia.org/T185232) (owner: 10Jayprakash12345) [22:02:49] (03PS1) 10Urbanecm: Add several domains of Ukraine government to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405550 (https://phabricator.wikimedia.org/T185399) [22:03:46] (03CR) 10jerkins-bot: [V: 04-1] Add several domains of Ukraine government to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405550 (https://phabricator.wikimedia.org/T185399) (owner: 10Urbanecm) [22:15:05] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3915209 (10Deskana) [22:59:55] RECOVERY - MegaRAID on analytics1038 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy [23:20:34] !log bootstrapping restbase2012-a - T184100 [23:20:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:20:50] T184100: Reprovision legacy Cassandra nodes into new cluster - https://phabricator.wikimedia.org/T184100 [23:59:54] PROBLEM - MegaRAID on analytics1038 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough