[00:17:11] 10Labs-project-Wikistats, 10Patch-For-Review: Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats - https://phabricator.wikimedia.org/T165879#3336732 (10Dzahn) @Xqt I will reactivate XML soon, i'll just run the mysql -X command as before but let it happen like once per hour i... [00:35:32] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:00:27] 10Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3336815 (10Betacommand) [01:00:31] 10Labs, 10DBA: Missing plwikisource_p.revision data on labs replica database - https://phabricator.wikimedia.org/T167554#3336814 (10Betacommand) [01:40:31] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [02:01:32] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:41:33] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [05:56:27] 10Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3336943 (10Ankry) The revisions missing from plwikisource_p were created between 20160428113148 and 20160428122541. page table entries created in the same period, eg. https://pl.wikisource.org/w/index.php?title=Specjalna:Wk%... [05:57:17] 10Labs, 10DBA: Missing plwikisource_p.revision data on labs replica database - https://phabricator.wikimedia.org/T167554#3336944 (10Ankry) The revisions missing from plwikisource_p were created between 20160428113148 and 20160428122541. page table entries created in the same period, eg. https://pl.wikisource.o... [05:57:52] 10Labs, 10DBA: Missing plwikisource_p.revision and plwikisource_p.page data on labs replica database - https://phabricator.wikimedia.org/T167554#3336945 (10Ankry) [07:43:53] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10cloud-services-team (Kanban): Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3336974 (10zhuyifei1999) I just remembered that [[https://en.wikipedia.org/wiki/Gentoo_Linux|Gentoo]] has a package... [09:08:39] 10Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: wsexport tool leaking files in /tmp - https://phabricator.wikimedia.org/T166337#3336994 (10Tpt) I have setup a cron task running daily that executes the following bash script: ``` rm /tmp/ws-* rm /data/project/wsexport/tool/temp/ws-* rm /data/project/wsexport/x... [10:41:49] 10Tool-Labs-tools-stewardbots: Let stewardbot display temporary rightchanges too - https://phabricator.wikimedia.org/T164204#3225412 (10Luke081515) [10:44:06] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3337035 (10zhuyifei1999) Yep I'm encountering this all the time. ``` NAME READY STATUS... [11:06:33] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337051 (10Kotz) I am cleaning up now. You may delete this files on whim. Most are very old and the tool should not be creating so much. I will track it going forward. thanks [12:55:14] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337111 (10Kotz) 05Open>03Resolved added scheduled deletion to crontab jsub -N deltemp -once -quiet find /tmp -iname "*hasava*" -user tools.bothasava -mtime 14 -type f -delete [14:23:41] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:58:42] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:05] 10Tool-Labs-tools-Other: templatecount is displaying very large error messages - https://phabricator.wikimedia.org/T163178#3337352 (10Jc86035) 05Open>03Resolved a:03Jc86035 Separately reported [[ https://github.com/Krinkle/intuition-web/issues/2 | on Github ]]. [17:22:32] 10Labs-project-Wikistats, 10Patch-For-Review: Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats - https://phabricator.wikimedia.org/T165879#3337404 (10Xqt) Great. [17:46:03] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3336196 (10zhuyifei1999) >>! In T167542#3337111, @Kotz wrote: > jsub -N deltemp -once -quiet find /tmp -iname "*hasava*" -user tools.bothasava -mtime 14 -type f -delete `jsub` e... [17:48:14] 10Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: wsexport tool leaking files in /tmp - https://phabricator.wikimedia.org/T166337#3293050 (10zhuyifei1999) >>! In T166337#3336994, @Tpt wrote: > I have setup a cron task running daily that executes the following bash script: Note: Make sure they are executed on `t... [17:52:18] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337449 (10Kotz) @zhuyifei1999 I understand. I use `cron` to run the bot periodically with `jsub`. I was afraid it was creating too many tmp files in there. maybe it was wrong... [18:33:53] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337477 (10zhuyifei1999) >>! In T167542#3337449, @Kotz wrote: > Is /tmp saved between invocations on the grid? Yes and no. There are several things to consider here: * If the te... [18:38:11] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337479 (10Kotz) @zhuyifei1999 thanks The bot needs to save some caching information which is best to be available across invocations. Is the best place to save it would be /... [18:43:43] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337485 (10zhuyifei1999) Yes, if you need the data saved across invocations, `/data/project/` can be a choice, although [[https://wikitech.wikimedia.org/wiki/Help:Shared_storage#... [18:59:02] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3337490 (10Kotz) OK. I moved the cache file creation to /data/project/bothasava/ . thanks @zhuyifei1999 [23:06:21] gah, I can'T remeber it. What was the url of the tool or page to browse openstack projects? (not horizon) [23:08:07] Sagan https://tools.wmflabs.org/openstack-browser/ [23:08:13] ah, thx [23:08:44] your welcome :)