[01:04:50] 10Differential, 10Phabricator, 10Developer-Wishlist (2017): Automatically add "patch-for-review" tag when `arc diff` - https://phabricator.wikimedia.org/T150510#2788267 (10srishakatux) @Volker_E we will begin adding proposals to a MediaWiki page for the developer wishlist's voting phase soon, could you elabo... [02:06:54] 10Differential, 10Phabricator, 10Developer-Wishlist (2017): Automatically add "patch-for-review" tag when `arc diff` - https://phabricator.wikimedia.org/T150510#2988663 (10Volker_E) [06:47:21] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2989006 (10Marostegui) db2012 caught up nicely so I believe this ticket can be closed. We can disc... [06:48:21] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2989008 (10Marostegui) 05Open>03Resolved a:03jcrespo [09:27:49] 10Phabricator, 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2989246 (10Nikerabbit) Now it is also happening for the search box that is on top of every page. It seems to me that there is some queue system,... [10:05:17] 10Phabricator, 10DBA: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) [10:06:45] 10Phabricator, 10DBA: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989326 (10Marostegui) [10:08:15] 10Phabricator, 10DBA: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) ``` check_ferm invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0 check_ferm cpuset=/ mems_allowed=0-1 CPU: 9 PID: 29001 Comm: check_ferm Tainted: G I 4.4.0-2-amd64 #1... [10:13:14] 10Phabricator, 10DBA: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Volans) Looks like there was some heavy load on the server in the ~20 minutes before the OOM: https://grafana.wikimedia.org/dashboard/file/server-board.json?var-server=db1043&var-network=eth0&from=148594039... [10:19:12] 10Phabricator, 10DBA: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989362 (10Marostegui) There are some really slow queries running at the time: https://tendril.wikimedia.org/report/slow_queries?host=%5Edb1043&user=&schema=&qmode=eq&query=&hours=1 The replicat (db1048 also crashed a... [10:31:51] 10Phabricator, 10DBA: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989371 (10Marostegui) This is the last insert we got before the crash (of both, master and slave) ``` INSERT INTO `search_documentfield` (phid, phidType, field, auxPHID, corpus, stemmedCorpus) VALUES ('PHID... [10:33:02] 10Phabricator, 10DBA: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989373 (10Marostegui) p:05Triage>03High [10:34:17] 10Phabricator, 10DBA: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) I have tried to reproduce that insert on the following versions on my lab: - 10.0.23 - 10.0.28 And they didn't crash (just checking if has something to do with: T156373) [10:57:49] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) [11:12:05] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989405 (10Marostegui) >>! In T156905#2989343, @Volans wrote: > Looks like there was some heavy load on the server in the ~20 minutes before the OOM: > > https:/... [11:29:50] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989430 (10Paladox) @Marostegui hi, it looks like the server is running version Server version: 10.0.23-MariaDB-log Should we update it to 10.0.29 the package th... [11:38:43] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989448 (10Paladox) @Marostegui could it b a memory leak? could this https://github.com/MariaDB/server/commit/b7dc830 be the fix? [11:46:20] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989466 (10jcrespo) > However, db1048 (the slave) also crashed and that one is only supposed to have the replication thread running right? No, at around 09:39:31... [11:47:21] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989467 (10Marostegui) >>! In T156905#2989430, @Paladox wrote: > @Marostegui hi, it looks like the server is running version Server version: 10.0.23-MariaDB-log >... [11:51:48] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989474 (10Paladox) does this mean that phabricator needs improvements to it's query? [12:52:50] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989591 (10Marostegui) >>! In T156905#2989474, @Paladox wrote: > does this mean that phabricator needs improvements to it's query? I believe so, or at least on t... [14:14:23] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989711 (10epriestley) I'm not sure why the query is slow or requires a significant amount of memory. The structure of the query specifically tries to avoid this,... [14:16:55] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989717 (10epriestley) One vague possibility is that you may have a few documents which are extremely large: for example, perhaps someone wrote a 1GB comment on a... [14:22:13] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989733 (10jcrespo) [14:22:29] 10Phabricator, 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2989740 (10Aklapper) >>! In T143863#2989246, @Nikerabbit wrote: > Now it is also happening for the search box that is on top of every page. It se... [14:23:19] andre__: possible botspam? https://phabricator.wikimedia.org/T156915#2989689 [14:37:22] 10Phabricator, 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2989812 (10Nikerabbit) I see this is also being discussed in T156905#2989711. And no, the link you gave did not load for me in a reasonable time. [14:46:32] TabbyCat: andre__ I looked at the user and linked accounts etc and decided to disable [14:46:50] chasemp: good, thanks! [14:47:02] I realize it's aggressive after first instance of bad behavior but considering recent history I don't have time to babysit and they can appeal [14:47:12] and nothing about that account seems legit [14:47:30] we've got some spamming recently [14:47:34] 10Phabricator: Improve the detection of archived projects with tasks open that have no other projects assigned - https://phabricator.wikimedia.org/T136695#2989856 (10Aklapper) 05stalled>03declined a:05Danny_B>03None Unfortunately closing this report as no further information has been provided. @Danny_B:... [14:47:45] I've reported quite a bnch of those already to André [14:49:51] looking at the account IP can also be revealing. If it comes from the usual suspect countries it's most likely a spambot [14:52:18] thanks! [14:52:44] (We also get one file upload per day with copyrighted material but from very different IPs. I do check regularly...) [14:53:14] same pattern as https://phabricator.wikimedia.org/p/Locat32/ [14:53:19] (already dealt with) [14:53:46] Oh, that user again :P [14:54:03] same as https://phabricator.wikimedia.org/p/Lucombo/ and such [14:57:29] 10Differential, 10Phabricator, 10Developer-Wishlist (2017): Automatically add "patch-for-review" tag when `arc diff` - https://phabricator.wikimedia.org/T150510#2989877 (10Aklapper) >>! In T150510#2988288, @greg wrote: > If the Differential revision has an associated task then it automatically shows up ...an... [15:10:00] andre__: seemed like account approval wasn't too bad as far as volume (although I still get all those emails :) [15:10:07] maybe we should switch for a week or so idk [15:11:21] chasemp, it was okayish. I'm more wondering about the long-term and if things happen again. Hence my question about talking to upstream... [15:11:33] yup [15:11:43] they always say they want proof positive of actual vandals as an issue :D [15:11:50] well, we are here to oblidge [15:11:57] oblige even [15:26:37] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989959 (10Marostegui) Hey @epriestley - thanks for the fast response! >>! In T156905#2989711, @epriestley wrote: > I'm not sure why the query is slow or requi... [15:34:52] 10Differential, 10Phabricator, 10Developer-Wishlist (2017): Automatically add "patch-for-review" tag when `arc diff` - https://phabricator.wikimedia.org/T150510#2788267 (10epriestley) To provide a little upstream perspective: we don't currently have support for this because we haven't seen (much? any?) inter... [15:41:02] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989990 (10Marostegui) >>! In T156905#2989959, @Marostegui wrote: > > ``` > | 50776 | root | localhost | phabricator_search | Query | 605 |... [15:42:41] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989991 (10epriestley) Hrrm. For comparison, a similar inner query on `secure.phabricator.com` (searching for "phabricator" instead of "affecting translatewiki.ne... [15:51:59] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990013 (10Paladox) What if we dropped the phabricator_search table then re created and then do the reindexing for mysql, would that work? [15:56:33] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990019 (10Marostegui) >>! In T156905#2989991, @epriestley wrote: > Hrrm. For comparison, a similar inner query on `secure.phabricator.com` (searching for "phabri... [15:58:54] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990020 (10epriestley) Here's another possible formulation of the query based on Googling "FULLTEXT initialization", although I haven't yet found a real descripti... [16:02:25] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990021 (10jcrespo) Maybe tuning `innodb_ft_cache_size` and `innodb_ft_total_cache_size` is needed. Little to no tuning was done after converting the table from M... [16:03:58] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990030 (10epriestley) Ah! That seems pretty broken -- the same simple query takes `0.03s` on `secure.phabricator.com`, so yours is ~22,000x slower for ~10x more... [16:06:25] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990034 (10jcrespo) > Is that something you're comfortable trying? Yes, that makes lot of sense, too. [16:07:38] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990035 (10epriestley) Tuning `innodb_ft_*` parameters may also be fruitful, but I don't have any direct experience with it to provide guidance. [16:08:30] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990036 (10Marostegui) >>! In T156905#2990030, @epriestley wrote: > Ah! That seems pretty broken -- the same simple query takes `0.03s` on `secure.phabricator.com... [16:24:54] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990100 (10epriestley) If we can't get the trivial `SELECT * WHERE MATCH(...)` case performing quickly, I believe the engine isn't going to be usable no matter ho... [16:35:45] 10Phabricator, 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2990117 (10Nemo_bis) `curl -L https://phabricator.wikimedia.org/maniphest/query/qHfJ8C2aAHVV > /dev/null` still got no data for me after 5 minut... [16:36:01] 10Phabricator, 10Developer-Wishlist (2017), 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2990118 (10Nemo_bis) [16:42:51] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990129 (10Paladox) We have elastic search as an experimental option. You can enable it through the pref by going to Developer Settings and under the elastic sear... [16:43:41] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990131 (10Marostegui) Good news: ``` root@MISC m3[phabricator_search]> optimize table search_documentfield; Stage: 1 of 2 'copy to tmp table' 0.023% of stage d... [16:45:57] 10Phabricator, 06Project-Admins, 10MediaWiki-General-or-Unknown, 10Developer-Wishlist (2017): Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?) - https://phabricator.wikimedia.org/T76942#2990149 (10Nemo_bis) [16:46:24] 10Phabricator, 10Developer-Wishlist (2017), 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2990152 (10Aklapper) >>! In T143863#2989812, @Nikerabbit wrote: > I see this is also being discussed in T156905#29... [16:46:31] 10Phabricator, 10Developer-Wishlist (2017), 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2990153 (10Aklapper) >>! In T143863#2990117, @Nemo_bis wrote: > `curl -L https://phabricator.wikimedia.org/maniph... [16:48:19] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990173 (10Marostegui) And the big query finished in `5 rows in set (1 min 59.41 sec)`: ``` root@MISC m3[phabricator_search]> SELECT documentPHID, MAX(fieldScore... [16:49:07] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990174 (10Paladox) but our only problem with elasticsearch is you can't reindex both indexes. So elasticsearch index is out of date. we need someway to be able... [16:49:13] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990175 (10Nemo_bis) >>! In T156905#2989343, @Volans wrote: > Looks like there was some heavy load on the server in the ~20 minutes before the OOM: > > https://g... [16:49:33] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990176 (10jcrespo) Let's still try to tune the innodb parameters. 2 minutes is also too much for a simple search. [16:51:57] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990178 (10Paladox) >>! In T156905#2990175, @Nemo_bis wrote: >>>! In T156905#2989343, @Volans wrote: >> Looks like there was some heavy load on the server in the... [17:04:26] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990225 (10Marostegui) >>! In T156905#2990176, @jcrespo wrote: > Let's still try to tune the innodb parameters. 2 minutes is also too much for a simple search. A... [17:54:42] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990460 (10Paladox) >>! In T156905#2989959, @Marostegui wrote: > Hey @epriestley - thanks for the fast response! > > > >>>! In T156905#2989711, @epriestley wro... [18:05:36] 10Phabricator, 07Availability, 13Patch-For-Review, 07WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#2990513 (10jcrespo) [18:14:13] 10Phabricator, 06Discovery, 06Discovery-Search, 10Elasticsearch, and 2 others: Setup a private elasticsearch cluster for phabricator - https://phabricator.wikimedia.org/T156939#2990532 (10Paladox) [18:15:05] 10Phabricator, 06Discovery, 06Discovery-Search, 10Elasticsearch, and 2 others: Setup a private elasticsearch cluster for phabricator - https://phabricator.wikimedia.org/T156939#2990550 (10Paladox) p:05Triage>03High Changing to high since the db keeps crashing due to full text indexes. T156905 [18:16:57] 10Phabricator, 06Discovery, 06Discovery-Search, 10Elasticsearch, and 2 others: Setup a private elasticsearch cluster for phabricator - https://phabricator.wikimedia.org/T156939#2990556 (10Paladox) [18:17:56] 10Phabricator, 06Discovery, 06Discovery-Search, 10Elasticsearch, and 2 others: Setup a private elasticsearch cluster for phabricator - https://phabricator.wikimedia.org/T156939#2990532 (10Paladox) Also per T155299#2975104 suggestion there. [19:01:01] phab seems down. known? [19:01:07] known see -ops [19:01:16] thx [19:27:48] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990866 (10Marostegui) This has happened again and crashed the master and the slave. I have run the ALTER table to optimize the table on the master while we were... [19:29:28] we're back ^ [19:30:03] 10Phabricator: Ability to move multiple workboards on same comment - https://phabricator.wikimedia.org/T156953#2990873 (10Matthewrbowker) [19:30:06] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990885 (10Paladox) p:05High>03Unbreak! Due to it taking down the db + forcing us to switch to elastic search. I am upping the priority to unbreak. [19:32:43] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990912 (10greg) p:05Unbreak!>03High Issue has been mitigated (we're using the ES backend for everyone now), lowering priority but keeping open for any follow... [19:50:08] 10Phabricator, 10Developer-Wishlist (2017), 07Performance: In "Edit parent tasks" dialog, searching "all open objects" for some text takes a long time - https://phabricator.wikimedia.org/T143863#2581485 (10greg) ElasticSearch is now default for everyone (as of a little bit ago). Let us know if this is still... [20:15:23] 06Project-Admins, 10Adminbot: Rename #adminbot to #stashbot? - https://phabricator.wikimedia.org/T156929#2991134 (10MarcoAurelio) I'd rename because it's simple and in any case we can add as additional hashtags the names of the old projects so they don't get 'orphan'. In case old bugs need fixing, a workboard... [20:59:02] 06Project-Admins, 10Striker: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2991299 (10Legoktm) OK, let's go with "Tool-{toolname}", for consistency with the Diffusion setup, e.g. . It's also... [22:01:48] 10Phabricator, 06Operations, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991483 (10Dzahn) [22:02:09] 10Phabricator, 06Operations, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991485 (10Dzahn) [22:02:12] 10Phabricator, 06Operations, 06Release-Engineering-Team: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#2839436 (10Dzahn) [22:04:38] 10Phabricator, 06Operations, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991492 (10Dzahn) The advantages here (over reinstalling iridium): - less maintenance/downtime for end users - don't have to fix everything at once in... [22:06:55] 06Project-Admins, 10Adminbot: Rename #adminbot to #stashbot? - https://phabricator.wikimedia.org/T156929#2991499 (10bd808) >>! In T156929#2991134, @MarcoAurelio wrote: > In case old bugs need fixing, a workboard col. specially for those could be created or a milestone inside StashBot, which we can archive whe... [22:10:43] 10Phabricator, 06Operations, 06Release-Engineering-Team, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991504 (10Dzahn) procurement ticket for iridium was https://rt.wikimedia.org/Ticket/Display.html?id=6772 they (12 misc s... [22:10:47] 10Phabricator, 06Operations, 06Release-Engineering-Team, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991506 (10Paladox) [22:21:42] 10Phabricator, 06Operations, 06Release-Engineering-Team, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991554 (10RobH) a:03mark iridium's (current eqiad phab host) specs: * 1U system * Dual Intel Xeon CPU E5-2450 v2 @ 2.5... [22:24:32] 10Phabricator, 06Operations, 06Release-Engineering-Team, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#2991562 (10mmodell) [22:33:25] 06Project-Admins, 10Adminbot: Rename #adminbot to #stashbot? - https://phabricator.wikimedia.org/T156929#2991591 (10Krinkle) +1 to rename. [22:43:14] 10Phabricator, 07Availability, 13Patch-For-Review, 07WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#2991631 (10Dzahn) [22:43:16] 10Phabricator, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Setup test domain for phab2001 - https://phabricator.wikimedia.org/T152132#2991626 (10Dzahn) 05stalled>03declined see reason above, we can't do this yet. Instead we are requesting new hardware for phab1001 (T156970) to test wi... [22:56:19] heh, speaking of stashbot: [22:56:21] 07:06 ~ stashbot_ is now stashbot__ [22:56:21] 07:07 ~ stashbot__ is now stashbot___ [22:56:21] 07:07 ~ stashbot___ is now stashbot____ [22:56:21] 07:08 ~ stashbot____ is now stashbot_____ [22:56:24] 07:08 ~ stashbot_____ is now stashbot______ [22:56:26] 07:09 ~ stashbot______ is now stashbot_______ [22:56:29] 07:09 ~ stashbot_______ is now stashbot________ [23:01:11] lol [23:01:34] maybe we should teach stashbot to /ns ghost [23:20:29] documented the process of adding forms to your "favorites" menu in phabricator: https://www.mediawiki.org/wiki/Phabricator/Forms [23:21:45] I'm not sure where I should link it from ... [23:48:01] 10Phabricator, 10DBA, 06Operations, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2991826 (10jcrespo) removing m3 from dbstore2001: db1043-bin.001457:753455796