[00:23:11] i just hacked my local /etc/hosts and works now [10:00:59] !log tools building & publishing toollabs-webservice 0.40 deb, and all Docker images T156626 T148872 T158244 [10:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:01:06] T156626: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626 [10:01:06] T158244: Improve `webservice status` output - https://phabricator.wikimedia.org/T158244 [10:01:07] T148872: Make webservice command read default cli arguments from ~/.webservicerc - https://phabricator.wikimedia.org/T148872 [13:17:08] is it normal that it takes 6½ minutes to drop a table in a tool database? [13:17:24] I was about to ask “is it normal that it just doesn’t complete” when it finally completed :) [13:17:37] but the table had only a single row, so I wouldn’t have expected it to take that long… [13:18:29] WikidataFacts: your table may be locked, it should only take a few seconds at most [13:18:54] hm, okay [13:19:09] for example, if you or other people are reading from the table, it cannot be dropped or altered until the other transasctions finish [13:19:25] perhaps my application had a lock on it (I was running the DROP TABLE from the command line) [13:19:27] it is a very typical problem called metadata locking [13:19:29] I’ll stop it next time, thanks [13:20:07] I can help you debug next time, or you can check your own processes if you are the only user with access to the table [13:20:59] more infor here: https://dev.mysql.com/doc/refman/5.5/en/metadata-locking.html [13:22:21] let’s see how many more times I have to drop the table :) [13:26:39] there is someone doing heavy usage of the database at the moment, I will research too [13:37:50] WikidataFacts: so probably not you [13:37:56] https://phabricator.wikimedia.org/T188205#4488332 [13:38:15] that was causing 100% cpu over the 8 cores [13:38:23] ah, okay [13:38:31] thanks for reporting [13:38:36] I’m getting much better response times now, thank you [13:38:43] such heavy usage is nor normal at all [13:38:45] (though I also stopped the webservice now, just in case it held a lock) [13:39:19] but the metadata thing is still real- if you are reading at the same time than an alter/drop table, you will lock yourself [13:39:24] so that shoul be helpful, too [13:40:04] the third thing is: https://grafana.wikimedia.org/dashboard/db/mysql?panelId=11&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1005&var-port=9104 [13:40:18] when it reaches near 0, all people will get better write performance [13:40:41] there was a spike tonight [14:00:42] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @leszek_wmde & @jakob_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:38] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @leszek_wmde & @jakob_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:38] (03PS1) 10MacFan4000: Updates [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/451340 [14:51:15] (03CR) 10MacFan4000: [V: 032 C: 032] Updates [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/451340 (owner: 10MacFan4000) [16:49:19] is there a shared directory on Toolforge where everyone can create files and everyone can also delete those files? [16:49:53] I’m thinking about adding some “kill switch” to a tool of mine – if e. g. /data/scratch/foobar exists, don’t start any expensive actions [16:50:12] but in /data/scratch, when someone else creates the kill switch, I can’t delete it, if I understand the permissions correctly [16:50:17] which would be inconvenient ^^ [16:50:59] (I could make the tool look at a different file, of course, but then I’d have to update all documentation pointing admins to that file) [16:54:59] hm, perhaps if I create an rwxrwxrwx (but not sticky-bit) directory within /data/scratch? [16:57:41] yup, that works, problem solved :) [17:07:32] WikidataFacts: you can override sticky bit ifyou own the containing directory [17:07:40] see unlink(2) [17:08:38] well, if I create a world-writable directory, I don’t even need the sticky bit, right? [17:08:48] if I own the directory, I’m allowed to delete everything in it [17:09:06] but perhaps adding the sticky bit would be useful to make sure no one *else* can remove files there? [17:09:25] if you create a world-writable directory without sticky, others can delete files in it [17:09:28] yes [17:09:33] ok [17:09:43] I need to think about whether I want that or not [17:09:45] but thanks :) [17:09:55] k np :) [17:49:34] i would like to convert the "ircecho" class to use systemd::service, trying to check what is using ircecho module, i see shinken does. do you know what distro that is on [17:50:04] should be ok to only support systemd and not sysvinit anymore for the irc bots? [18:31:01] mutante: per https://tools.wmflabs.org/openstack-browser/project/shinken it looks like the current shinken server is still Trusty [18:31:56] bd808: ah, thanks. ok [18:32:01] I think K.renair has been asking recently what we should do about that host/service. [18:32:18] and our answer has been a solid "good question!" [18:33:24] yea.. ehm.. honestly it would simplify puppet code if we could use icinga for this too [18:33:51] like we have those 3 modules, icings, shinken and the common monitoring one [18:34:02] and they are interconnected [18:34:26] the pick of shinken over icinga for Cloud things was before my time. I'm nto sure what the reasons were then and if they are still valid now [18:34:47] and then there is icinga2 as well and paladox made a module for miraheze and suggested using it [18:35:07] at one point icinga1 is going to be too old [18:35:15] yeah... I don't want to touch that discussion. Convince F.aidon first [18:35:20] then the question is if we will go to icinga2 [18:35:26] heh, fair enough, yea [18:35:43] just saying :) right now i just wanted to do puppet refactoring to use systemd::service [18:36:03] but yea [18:37:06] meanwhile we have networking issues in eqiad but people are looking [18:37:56] we will need to kill off Trusty by April 2019. The upgrade of the shinken project could happen as soon as 'someone' has time and I guess that would unblock your ircecho stuff [18:38:24] finding someone with time and skill to do the migration is the hard part I guess [18:39:42] ok, yes, i understand [18:48:08] * bd808 awards zhuyifei1999_ a barnstar for helping with the webservice package [18:48:23] * zhuyifei1999_ :) [18:48:40] wow yes, that might have made it so our gsoc student can do the last piece of her project, which would be awesome [22:00:38] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [22:02:54] addshore: your bot has gone rogue! Or at least does not understand the 23:00 UTC meeting only being once a month [22:50:54] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting