[10:40:57] (03PS9) 10Jean-Frédéric: Add ESlint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378011 (https://phabricator.wikimedia.org/T175907) [10:41:35] (03CR) 10Jean-Frédéric: "Thanks for picking up that patch :) I bumped grunt-eslint to latest" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378011 (https://phabricator.wikimedia.org/T175907) (owner: 10Jean-Frédéric) [13:44:40] !log wikispeech Deploy latest from Git master: 01f254d (T175810), cdda3ee, e559d75, 551ec23 (T177256), 6993ef4 (T172523), 8824cf8, 3dd8daf (T161097), 1929c06 [13:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [13:44:47] T177256: Decide whether we want to move phpcs.xml to .phpcs.xml - https://phabricator.wikimedia.org/T177256 [13:44:47] T175810: Add lexicon tool special page in Wikispeech extension - https://phabricator.wikimedia.org/T175810 [13:44:48] T172523: Add validation to input json in API - https://phabricator.wikimedia.org/T172523 [13:44:48] T161097: Recitation should include the article title - https://phabricator.wikimedia.org/T161097 [14:35:21] (03Draft1) 10Paladox: Fix syntax that is now incompatible with icingaweb2 2.5+ [labs/icinga2] - 10https://gerrit.wikimedia.org/r/393769 [14:35:24] (03Draft2) 10Paladox: Fix syntax that is now incompatible with icingaweb2 2.5+ [labs/icinga2] - 10https://gerrit.wikimedia.org/r/393769 [14:36:01] (03CR) 10Paladox: [V: 032 C: 032] Fix syntax that is now incompatible with icingaweb2 2.5+ [labs/icinga2] - 10https://gerrit.wikimedia.org/r/393769 (owner: 10Paladox) [17:42:20] !log deployment-prep Remove stale ORES customizations for the beta cluster. [17:42:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:31:35] hello [18:32:07] after forcing my tool to fail with the following error, no error.log file gets created [18:32:07] hello Superyetkin [18:32:08] PHP Fatal error: Call to undefined function xyz() in /mnt/nfs/labstore-secondary-tools-project/superyetkin/public_html/test.php on line 3 PHP Stack trace: PHP 1. {main}() /mnt/nfs/labstore-secondary-tools-project/superyetkin/public_html/test.php:0 [18:32:27] I just deleted my old error.log file with "rm" [18:33:31] Superyetkin: there are known issues with the log files not being recreated when they are deleted while the webservice is running [18:33:57] so you probably need to `webservice restart` to make a new error log show up [18:34:32] I know that grid engine has this problem at least. I don't remember if kubernetes handles it better [18:35:30] Superyetkin: was this to wipe the log contents? fyi 'truncate -s 0 ' [18:35:32] should work [18:35:46] but you still have to restart the process to release the handle for size reduction [18:36:07] yes, I intended to wipe error history [18:36:35] truncate will remove contents but not reduce size until teh holding process releases the file (seek) i.e. webservice restart [18:37:10] > foo or truncate -s 0 foo [18:37:18] rm sort of does the same thing, but the old inode becomes an orphan on the NFS server [18:37:35] its all ugly :/ [18:37:39] 'tis [18:38:05] hmm [18:38:15] after webservice restart, [18:38:25] I see the error.log file [18:38:25] webservice stop; ; webservice start; is the best plan for log rotation today [18:38:33] but the content is this [18:38:33] 2017-11-28 18:36:28: (log.c.166) server started [18:38:49] I triggered a PHP error but it does not show up there [18:39:21] it can take a minute or tow to show up because of NFS server lag/local caching [18:39:25] *two [18:39:38] let us wait [18:40:46] we have local file caching enabled on the NFS shares to help with the common case of reading source code, but that makes log files appear to lag [18:41:51] we had worse problems when we had the local cache disabled due to the number of round trips to the NFS server for starting new scripts [18:43:04] after five minutes, nothing is there... [18:44:26] Superyetkin: that does seem unusual. You triggered the fatal error again after restarting the webservice? [18:45:14] yes, I did [18:45:19] let me do it again [18:46:31] here is what I did [18:46:32] tools.superyetkin@tools-bastion-03:~$ webservice stop Stopping webservice. tools.superyetkin@tools-bastion-03:~$ rm error.log tools.superyetkin@tools-bastion-03:~$ webservice restart Your job is not running, starting. tools.superyetkin@tools-bastion-03:~$ php public_html/test.php PHP Fatal error: Call to undefined function xyz() in /mnt/nfs/labstore-secondary-tools-project/superyetkin/public_html/test.php on line 3 PHP [18:47:05] and no error displayed... [18:47:12] Superyetkin: that php fatal is from running at the command line, not from calling the webservice? [18:47:36] yes, from the command line [18:47:41] the error.log file is only the output of STDERR from the webservice process [18:47:47] I can access that script via HTTP [18:48:21] so, where is the PHP error log then? [18:48:46] for an interactive process the log is just what prints to your terminal [18:49:18] nothing is printed, the output is just blank... [18:49:26] no PHP error is displayed on the screen [18:49:27] the PHP Fatal error: ... [18:49:56] see this [18:49:57] tools.wmflabs.org/superyetkin/test.php [18:50:07] there is a syntax error there but the output is blank [18:50:27] that *should* write to error.log [18:50:39] but calling php on the command line should not [18:51:03] ok, the error has just arrived :) [18:51:04] the fatal error may be causing the process to crash before it actually logs anything however [18:51:09] in the error.log, I see that [18:51:12] thanks [18:51:29] but a glitch I recognized here [18:51:52] first, I RESTARTED the webservice and that did not work [18:52:18] after STOPPING the webservice, deleting ERROR.LOG and RESTARTING the webservice, it seemed to start working [18:53:10] ok, thanks for your help [18:53:26] interesting. I would expect a restart to work by itself, but maybe there is something happening there that prevents the error.log file from being repoened [18:53:29] *reopened [18:53:34] do you know a way to disable query caching on replica servers? [18:53:54] SQL caching [18:54:17] my tools, I believe, are shoeing outdated (cached) results [18:54:41] SELECT SQL_NO_CACHE ... FROM ...; should work I think [18:55:16] I'm not sure how large the query cache is, but I would not expect it to hold results for very long [18:55:23] more info at https://dev.mysql.com/doc/refman/5.7/en/query-cache-in-select.html [18:56:35] the query cache should respect replication changes. I really would not expect it to cause stale results [19:01:44] hmm, adding that did not change anything [19:02:45] there is no query cache on any wmf-managed database [19:02:45] perhaps the replicas themselves are lagged? [19:04:34] the heartbeat says no lag is present [19:04:59] https://tools.wmflabs.org/replag/ [19:05:02] there is only lag on c1, which we do not recomend using [19:05:16] I am on new servers [19:05:24] and it is not possible to contact except if you do it on purpose [19:05:32] but suffering from the lack of user databases there [19:05:44] using SQLite in the mentime [19:06:15] you have tools to create user databases [19:06:33] ? [19:06:51] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases [19:07:19] sqlite on NFS will be very slow [19:07:26] will that live through December 15th? [19:07:37] toolsdb? of course [19:07:38] you can make user dbs on tools.db.svc.eqiad.wmflabs [19:07:44] and it has redundancy [19:07:46] no collocated with the replica data is the thing [19:07:51] unlike the old ones [19:07:53] s/no/not [19:07:58] hmm, then there is a huge misunderstanding on my side [19:08:23] we are not removing the possiblity of having user databases [19:08:47] I presumed all of the user databases would go offline when the old servers did die [19:08:52] https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown -- "Migrate your user created tables to tools.db.svc.eqiad.wmflabs (also known as tools.labsdb) and JOIN via application space logic rather than in-process in the database." [19:08:53] we are just separating read-only replicas from writable (user) databases [19:09:05] Superyetkin: unless moved to toolsdb [19:09:25] you can literally copy them, import them to tools and that will be ok [19:09:43] we don't do that for you because we know there are many unused/abandoned ones [19:10:05] JOIN via application space logic rather than in-process in the database? [19:10:20] the catch is that IF you need to write and read replicas, you have to connect once to each server [19:10:26] my queries would work the other way around [19:10:36] JOIN within the SQL query [19:10:57] that is why I implemented SQLite for those [19:11:03] you create your own tables? [19:11:09] yes [19:11:21] I even have a namespace table :) [19:11:23] and do you read with the same application data from replicas? [19:11:41] I read data from replicas, yes [19:11:51] but not both on the same query [19:12:02] as you canot joing sqlite and mysql [19:12:08] of course [19:12:13] so this is the same [19:12:22] you CAN (it is not compulsory) [19:12:38] instead of use sqlite, create tables on mysql on toolsdb [19:12:55] we are going to keep user databases on toolsdb [19:13:00] unless my SQLite databases get too large, this would not be a problem, right? [19:13:02] those are not going away [19:13:15] that would be up to you [19:13:34] what about the road to the datasets_p schema? [19:13:46] I am looking forward to seeing it in action [19:14:05] I think there is not yet any solid decision, bd808? [19:14:32] for instance,why must every single user create a namespace table? [19:14:43] there should be only one [19:14:51] and everyone would use that [19:15:11] I said on the ticket, that precreating tables is something I am more than ok with [19:15:38] thanks jynus [19:15:39] but I am not the one to decide, it is a collective decision betwen what is desired and what it is possible [19:16:01] I am okay with that [19:16:22] and jcrespo is tring his hard, I think :) [19:16:22] feel free to comment on the ticket with your needs and/or opinion [19:16:36] you know I am jcrespo, right? [19:16:44] his comments on that topic was special [19:16:57] oh my :) [19:16:59] in what way? [19:17:19] I thought you were someone else [19:17:42] feel free to tell me how special I am :-) [19:18:02] ok jcrespo, I will revisit that ticket for sure [19:18:06] thanks for your time [19:18:42] have a good day [21:05:32] !help hi all, I'd like to download the last dump and use it on toolforge by my bot (that is on toolforge indeed)... what can I do? I've tried to wget it, but no enough space is available [21:05:32] avgas: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [21:05:59] avgas: dumps are already available to use from toolforge [21:06:02] avgas: the dumps are already available on toolforge, look in /public [21:06:10] :) [21:16:28] avgas: to be more specific, the xml dump files are HUGE. Toolforge provides a single local copy of all dumps for tools to share. See for example the files in /public/dumps/public/enwiki/20171103/ [21:18:07] avgas: there is a bit more information at https://wikitech.wikimedia.org/wiki/Help:Toolforge#Dumps [21:21:56] the best way to work with the shred dumps is using a library that can read them directly from the compressed files. halfak's MediaWiki XML library is one that I know works for this task -- https://github.com/mediawiki-utilities/python-mwxml [21:22:12] bd808: i know that information [21:22:16] * halfak peeks in [21:22:30] the mattes is that my /public/dumps directory is empty [21:22:46] avgas: ah! are you not on toolforge? [21:23:02] is this another project? [21:23:45] uhm [21:24:01] Or do we have a bug that is keeping the directory from mounting somewhere? [21:24:09] if I have access to bastion, I am on toolforge, isn't it? [21:24:30] avgas: what does `hostname -f` tell you? [21:24:45] bastion-01.bastion.eqiad.wmflabs [21:25:09] ok. that's a Cloud VPS bastion server and not a Toolforge server [21:25:23] the dumps files would not be mounted there [21:26:06] Try logging into tools-login.wmflabs.org instead [21:27:02] "Connection to tools-login.wmflabs.org closed by remote host. [21:27:02] Connection to tools-login.wmflabs.org closed." [21:27:59] (password are surely correct) [21:28:15] +*is [21:29:02] avgas: I'm tailing the auth.log there. can you try to ssh in again? [21:29:29] do we have any reason to think avgas is a member of tools? [21:29:56] avgas: what is your shell name? [21:30:02] and/or your username on wikitech? [21:30:09] avgas, are you the marco listed at https://tools.wmflabs.org/admin/tool/yacbot or someone else? [21:30:19] andrewbogott: now "avgas" is my shell name [21:30:34] old username "gnumarcoo" [21:30:51] (gnumarcoo was active on toolserver) [21:30:51] ah, so https://tools.wmflabs.org/admin/tool/gnubotmarcoo [21:31:00] thats correct [21:31:22] avgas: so you created a new/second account on wikitech? [21:31:27] ok so theres to change maintainers [21:31:33] from gnumarcoo to avgas [21:31:44] https://wikitech.wikimedia.org/wiki/User_talk:.avgas -- granted shell 2 years ago, but not a toolforge member [21:31:57] bd808: okok, now it's clear [21:32:20] yes, two account is active, i think [21:32:38] avgas: why two accounts? Isn't that going to lead to constant confusion (e.g. what's happening right now)? [21:32:46] but gnumarcoo is the old one so it can be closed [21:35:26] gnumarcoo seems to be renamed also on wikitech [21:36:17] no, it's just a page-redirect currently. All the old contributions are still attached. https://wikitech.wikimedia.org/wiki/Special:Contributions/Gnumarcoo [21:36:31] https://wikitech.wikimedia.org/w/index.php?title=User:Gnumarcoo&action=history [21:38:07] ok so according with you what can I do to fix? [21:39:04] you can either a) use your existing gnumarcoo account or b) request Toolforge membership for your avgas account [21:41:16] ok so I'll request Toolforge membership [21:41:21] thanks! [21:44:39] avgas: https://toolsadmin.wikimedia.org/tools/membership/apply is the place to ask :) [21:54:21] bd808: already done, thanks a lot [21:55:50] madhuvishy beat me to approving it :) [22:04:33] !log tools.admin Added Quiddity as a maintainer (T178661) [22:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:04:36] T178661: Drop wb_entity_per_page views in Wiki Replicas - https://phabricator.wikimedia.org/T178661 [22:04:51] that's not the right ticket number :/ [22:05:12] !log tools.admin Added Quiddity as a maintainer (T179511) [22:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:05:16] T179511: Add quiddity as toolsadmin in Striker - https://phabricator.wikimedia.org/T179511 [22:05:31] bd808: it usually takes me 9 times to get it right so 🤷‍♂️ [22:06:32] !log tools.admin Removed Ori.livneh from maintainers list [22:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:06:48] I corrected sal for ya [23:21:56] !log massmessage Created project with Noella94, Legoktm, and D3r1ck01 as administrators (T181373) [23:21:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Massmessage/SAL [23:22:03] T181373: Requesting a new project "MassMessage-Testing" on Cloud VPS for testing - https://phabricator.wikimedia.org/T181373 [23:23:12] :D