[00:04:38] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [00:20:54] !log rcm Neon: Restarted webservices to solve cache problem [00:20:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [00:43:43] !log rcm Xenon: Creating backup of the database [00:43:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [00:44:48] !log rcm CAC: Creating backup of the database [00:44:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [01:12:12] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:34:05] 10cloud-services-team, 10Goatification, 10User-bd808: Engage the unicorn community - https://phabricator.wikimedia.org/T173112#3520048 (10Rfarrand) [01:46:40] bd808: what's going on with all these goats and unicorns? [01:46:59] Probably an inside joke... [01:47:19] yeah I bet [01:51:42] I know the unicorns reference Labs. [01:51:46] Or whatever it's called now... [01:52:11] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [01:52:37] probably [01:52:54] what about cats and goats? [01:53:08] it says goats are replacing cats? [01:55:34] zhuyifei1999_: just having a bit of fun. :) the WMDE team have been using goats as cute pictures on slide decks [01:55:52] okay [01:56:57] How could this not make someone smile - https://commons.m.wikimedia.org/wiki/File:الأركان.JPG [01:58:48] * zhuyifei1999_ googles Goatification and sees Illuminati [01:59:28] https://twitter.com/goatification [02:13:13] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:27:09] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:53:12] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [03:02:10] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [03:49:52] (03PS1) 10Legoktm: Add configuration for `##goat` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/371599 [03:50:11] (03CR) 10Legoktm: [C: 032] Add configuration for `##goat` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/371599 (owner: 10Legoktm) [03:50:30] (03Merged) 10jenkins-bot: Add configuration for `##goat` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/371599 (owner: 10Legoktm) [03:50:48] (03CR) 10jenkins-bot: Add configuration for `##goat` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/371599 (owner: 10Legoktm) [03:52:48] !log tools.wikibugs Updated channels.yaml to: e3ddebf85e46ba8f8f39722492c5c9ca327edb1a Add configuration for `##goat` [04:11:12] (03PS1) 10Lokal Profil: Handle sparql http timeout issue [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/371602 (https://phabricator.wikimedia.org/T172690) [06:45:40] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:25:38] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:16] I'm doing some NFS-intensive operations on tools-bastion-05. might make puppet angry [08:05:18] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:35:17] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:06:44] 10Tools: Tool "wikitrends" loads cc-by buttons from creativecommons and licensebuttons.net - https://phabricator.wikimedia.org/T173068#3520319 (10Liuxinyu970226) >>! In T173068#3517740, @zhuyifei1999 wrote: > The tool maintainer is Johang / johan.gunnarsson. I'm unable to find their Phabricator username. @Johan ? [10:28:36] 10Tools: Tool "wikitrends" loads cc-by buttons from creativecommons and licensebuttons.net - https://phabricator.wikimedia.org/T173068#3520344 (10Johan) It's not me. (: (As far as I can tell, no Wikitrends Phab task has the maintainer subscribed to it. I'd assume he doesn't have a Phab account.) [14:04:24] 10Toolforge, 10Documentation, 10Kubernetes: Create visual diagram of documented components of Toolforge Kubernetes cluster - https://phabricator.wikimedia.org/T172855#3520534 (10Jprorama) Plan to use the [[ https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Node_naming_conventions | node n... [14:12:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:03:53] 10Tools: Tool "wikitrends" loads cc-by buttons from creativecommons and licensebuttons.net - https://phabricator.wikimedia.org/T173068#3520657 (10Liuxinyu970226) gomen [15:05:42] w/win 51 [15:13:12] 10Tools: Tool "traffic-grapher" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T172964#3520707 (10Liuxinyu970226) >>! In T172964#3514623, @zhuyifei1999 wrote: > The tool maintainer is KevinPayravi. I cannot find their Phabricator username. @SuperHamster By searching "Kevin Payravi" [15:15:15] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3520710 (10chasemp) > du -sh /srv/tools/shared/tools/project/admin/* ```2.2G /srv/tools/shared/tools/project/admin/access.log 0 /srv/tools/shared/tool... [15:17:06] bd808: I unfortunately cannot find your code https://github.com/search?q=bd808&type=Repositories&utf8=✓ [15:24:28] Nvm [15:25:36] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:28:21] 10Tools: Tool "traffic-grapher" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T172964#3520734 (10SuperHamster) a:03SuperHamster [15:28:44] 10Tools: Tool "traffic-grapher" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T172964#3514609 (10SuperHamster) Thanks for bringing this to my attention. Currently caught up in Wikimania but will fix this soon. [15:47:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [15:50:20] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3520776 (10Emijrp) [15:50:22] 10Tools: Tool "wlm-maps" loads tiles from OSM - https://phabricator.wikimedia.org/T173073#3520774 (10Emijrp) 05Open>03Resolved I have fixed the tiles links. It should be ok now. [16:05:39] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:50] Cyberpower678: are you looking for striker? [16:08:02] chasemp: ? [16:08:10] Cyberpower678: "bd808: I unfortunately cannot find your code" [16:08:18] chasemp: I found it. [16:09:29] https://github.com/bd808/toolforge-php-mysql [16:10:18] I should transfer that to https://github.com/toolforge [17:39:19] 10cloud-services-team, 10Goatification, 10Wikimania-Hackathon-2017, 10User-bd808: Engage the unicorn community - https://phabricator.wikimedia.org/T173112#3521095 (10Reedy) [17:57:38] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521142 (10chasemp) > /usr/local/sbin/logcleanup --config /etc/logcleanup-config.yaml --debug ```2017-08-12 17:56:41,618 DEBUG Namespace(config='/etc/... [18:10:55] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521168 (10chasemp) 2017-08-12 18:01:08,214 DEBUG /srv/tools/shared/tools/project/admin/access.log 2017-08-12 18:01:08,214 DEBUG /srv/tools/shared/tool... [18:17:35] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521171 (10chasemp) So python still thinks the original access.log is the 2.2G from pre rotation > /srv/tools/shared/tools/project/admin/access.log 22... [18:38:25] !log tools retart admin webservice [18:38:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:39:18] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521207 (10chasemp) du -b access.log 2288534827 access.log > webservice restart du -b access.log 2288568410 access.log [18:41:41] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521209 (10chasemp) > access.log tools.admin@tools-bastion-03:~$ du -b access.log 0 access.log sleep 30 du -b access.log 2288607847 access.log [18:49:46] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521214 (10chasemp) ```tools.admin@tools-bastion-03:~$ truncate --size 0 access.log tools.admin@tools-bastion-03:~$ du -b access.log 0 access.log tools... [18:54:56] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521216 (10chasemp) ```tools.admin@tools-bastion-03:~$ du -b access.log 2288823046 access.log tools.admin@tools-bastion-03:~$ ls -i access.log 80122951... [19:09:38] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521276 (10chasemp) wut ```root@labstore1004:~# du -b /srv/tools/shared/tools/project/admin/access.log 287995 /srv/tools/shared/tools/project/admin/ac... [19:10:42] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521284 (10chasemp) https://gerrit.wikimedia.org/r/#/c/346177/ [19:28:30] 10Tools, 10Wikisource: phetools: Sortable charts in Wikisource stats - https://phabricator.wikimedia.org/T173147#3521318 (10Aklapper) [19:31:27] 10Cloud-Services, 10Patch-For-Review: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#3521322 (10chasemp) > lighttpd 26283 tools.admin 5w REG 0,30 599081 80122951 /mnt/nfs/labstore-sec... [19:56:49] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3521358 (10bd808) [19:57:01] (03PS1) 10BryanDavis: Use new Toolforge logo and favicon [labs/striker] - 10https://gerrit.wikimedia.org/r/371645 (https://phabricator.wikimedia.org/T168480) [20:08:26] (03PS2) 10BryanDavis: Update static assets [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/364360 (https://phabricator.wikimedia.org/T149458) [20:29:16] !ping [20:29:22] !ping is pong [20:29:22] Key was added [20:47:06] Changed default instance of #wikimedia-cloud to goat-bot [20:47:06] @instance goat-bot #wikimedia-cloud [22:56:20] 10Data-Services, 10DBA: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3477438 (10Urbanecm) Poke. [23:20:16] are the stretch images stable now? [23:22:05] legoktm i think so [23:22:11] legoatm ^^ [23:53:01] bd808: we have https://github.com/toollabs/ and https://github.com/tool-labs/, now neither of them can use https://github.com/toolforge :(