[00:13:24] 10Tools: Tool "parliamentdiagram" loads assets from ajax.googleapis.com and github - https://phabricator.wikimedia.org/T172754#3508552 (10Slashme) OK, I'll do that in the next few days. [00:38:53] 10wikitech.wikimedia.org, 10Deployment-Systems, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3480850 (10mmodell) This is something I've been wanting to do for a long time. I find it difficult and annoying... [00:39:06] 10wikitech.wikimedia.org, 10Deployment-Systems, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3508606 (10mmodell) [00:41:13] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:07:40] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:16:11] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [01:37:11] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:12:12] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [02:42:41] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [03:55:55] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:20:55] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [06:08:32] (03PS1) 10Reception123: Add so that people know who to ask if they need ZppixBot in their channel [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370610 [06:09:13] 10Tools: Tool "editathonstat" loads assets from google - https://phabricator.wikimedia.org/T172517#3508829 (10Ranjithsiji) Thank You [06:34:49] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:09:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:31] 10Tool-Pageviews, 10Possible-Tech-Projects: Add Mediaviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#3508972 (10Richard_Nevell_WMUK) Thanks for putting this together, it will be very useful. And I hope the hackathon goes well! [10:56:26] 10Tool-Pageviews, 10Possible-Tech-Projects: Add Mediaviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#3509011 (10Qgil) @MusikAnimal if you plan to work on this task during the Hackathon (yay!) please add the tag #wikimania-hackathon-2017. [11:42:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:22:02] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [12:50:46] 10Cloud-Services: Unable to drop tables on dewiki.labsdb/s51072__dwl_p - https://phabricator.wikimedia.org/T172784#3509194 (10Giftpflanze) [13:12:01] 10Cloud-Services: Unable to drop tables on dewiki.labsdb/s51072__dwl_p - https://phabricator.wikimedia.org/T172784#3509230 (10jcrespo) a:03jcrespo I will try to repair the tables for you, but sadly aria/myisam are not very reliable storage engines and this can happen. [13:28:13] 10Cloud-Services, 10Cloud-VPS, 10Operations: logrotate/disk space on silver for nutcracker log - https://phabricator.wikimedia.org/T120683#3509264 (10Andrew) 05Open>03Resolved When all is well, silver is fine -- If something breaks that increases the logging rate then I get alerts. That said, I think we... [13:48:26] (03CR) 10Lokal Profil: "I've done done more on this and will submit a new patch after landing." [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [13:49:12] (03CR) 10Lokal Profil: [C: 04-1] Make unused image reports deal with sparql harvested data [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [14:25:00] there is something ongoing on labsdb replicas [14:25:06] https://tools.wmflabs.org/replag/ [14:25:14] I am trying to find out what exactly [14:30:50] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:32:22] PROBLEM - Puppet errors on tools-worker-1016 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:47:44] (03CR) 10Jean-Frédéric: [C: 032] build: bump grunt and drop grunt-cli [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/368765 (owner: 10Hashar) [14:48:01] (03CR) 10Jean-Frédéric: [C: 032] "Thanks for the maintenance effort Hashar :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/368765 (owner: 10Hashar) [14:48:38] (03Merged) 10jenkins-bot: build: bump grunt and drop grunt-cli [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/368765 (owner: 10Hashar) [14:49:25] (03CR) 10Jean-Frédéric: [C: 032] Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) (owner: 10Lokal Profil) [14:49:27] (03CR) 10jenkins-bot: build: bump grunt and drop grunt-cli [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/368765 (owner: 10Hashar) [14:50:16] (03Merged) 10jenkins-bot: Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) (owner: 10Lokal Profil) [14:51:05] (03CR) 10jenkins-bot: Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) (owner: 10Lokal Profil) [15:01:28] 10Cloud-Services, 10DBA: Unable to drop tables on dewiki.labsdb/s51072__dwl_p - https://phabricator.wikimedia.org/T172784#3509557 (10jcrespo) There seems to be corruption on allmost all, if not all Aria table, it seems to have internally crashed, not allowing even dropping the table. After I delete physically... [15:07:24] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [15:10:50] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [15:14:19] 10Tools, 10Patch-For-Review: Tool "fastilybot" loads assets from google and bootstrapcdn - https://phabricator.wikimedia.org/T172524#3509589 (10Ricordisamoa) https://github.com/fastily/fastilybot-toollabs/pull/1 [16:07:14] 10Tools, 10InternetArchiveBot: Tool "iabot" loads assets from google and hotjar - https://phabricator.wikimedia.org/T172605#3509743 (10Cyberpower678) p:05Triage>03Normal [16:25:22] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:44:13] 10Cloud-Services, 10DBA: Unable to drop tables on dewiki.labsdb/s51072__dwl_p - https://phabricator.wikimedia.org/T172784#3509834 (10jcrespo) @Giftpflanze It should work now, Aria had crashed internally, even if MySQL was still up. My advice to avoid this issue in the future is to convert/create the tables to... [16:44:19] labsdb issues should be fixed now [16:45:30] 10Cloud-Services, 10DBA: Unable to drop tables on dewiki.labsdb/s51072__dwl_p - https://phabricator.wikimedia.org/T172784#3509835 (10jcrespo) @marostegui labsdb1003 misbehaved today, I tried to fix it without rebooting, but it ended up crashing and recovering itself to a better state. I have migrated some key... [16:47:36] replicas are currently catching up [16:51:01] jynus this is a good thing... anyway do you have a rough estimate until labsdb will be completely caught up? [16:54:53] Zppix: it should take less than what was delayed https://tools.wmflabs.org/replag/ [16:55:00] maybe an hour or so? [16:55:18] only one server had issues [16:55:30] you can check all shards on the other if you need them now [16:56:02] jynus i dont i was just curious thanks :) [16:59:59] Not very important but is there any explanation why even though I'm a maintainer for both irclogbot and ZppixBot I'm not listed as a maintainer here: http://tools.wmflabs.org/admin/tools? [17:00:19] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:13] bd808 ^^ [17:17:00] (03PS1) 10Reception123: Added breaks as now everything displayed on one line [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370669 [17:19:51] (03CR) 10Zppix: [C: 032] Add so that people know who to ask if they need ZppixBot in their channel [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370610 (owner: 10Reception123) [17:19:58] (03Merged) 10jenkins-bot: Add so that people know who to ask if they need ZppixBot in their channel [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370610 (owner: 10Reception123) [17:20:01] (03CR) 10Zppix: [C: 032] Added breaks as now everything displayed on one line [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370669 (owner: 10Reception123) [17:20:07] (03Merged) 10jenkins-bot: Added breaks as now everything displayed on one line [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370669 (owner: 10Reception123) [17:26:01] 10Tool-Zppixbot, 10Documentation, 10User-Zppix: Write documentation for ZppixBot - https://phabricator.wikimedia.org/T149500#3509926 (10Zppix) [17:27:35] 10Cloud-Services, 10Documentation: Update documentation for the various phab Cloud tags - https://phabricator.wikimedia.org/T129043#3509929 (10Zppix) [17:27:46] 10Cloud-Services, 10Documentation: Update documentation for the various phab Cloud tags - https://phabricator.wikimedia.org/T129043#2093206 (10Zppix) [17:28:04] 10Cloud-Services, 10Documentation: Update documentation for the various phab Cloud tags - https://phabricator.wikimedia.org/T129043#2093206 (10Zppix) [17:28:23] sorry for spam [17:38:12] 10Tool-Zppixbot, 10Documentation, 10User-Zppix: Write documentation for ZppixBot - https://phabricator.wikimedia.org/T149500#3509988 (10Zppix) a:03Reception123 [17:48:32] 10Cloud-Services, 10DBA: Unable to drop tables on dewiki.labsdb/s51072__dwl_p - https://phabricator.wikimedia.org/T172784#3510005 (10Giftpflanze) 05Open>03Resolved I went through my table creation statements and added engine statements. I dropped four tables that I want to recreate and changed the engine o... [18:03:57] (03PS1) 10Reception123: Add links for release tag and my talkpage. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370675 [18:04:35] (03PS2) 10Reception123: Add links for release tag and my user page. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370675 [18:05:18] (03PS3) 10Reception123: Add links for release tag and my user page. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370675 [18:07:11] (03PS4) 10Zppix: Add links for release tag and my user page. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370675 (owner: 10Reception123) [18:07:19] (03CR) 10Zppix: [C: 032] Add links for release tag and my user page. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370675 (owner: 10Reception123) [18:07:24] (03Merged) 10jenkins-bot: Add links for release tag and my user page. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370675 (owner: 10Reception123) [18:27:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:39:25] (03PS1) 10Reception123: New release (v2) of ZppixBot [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370679 [19:02:37] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [19:18:37] 10Tools: Tool "parliamentdiagram" loads assets from ajax.googleapis.com and github - https://phabricator.wikimedia.org/T172754#3510235 (10Slashme) 05Open>03Resolved Done! [19:18:43] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3510237 (10Slashme) [19:26:56] (03CR) 10Zppix: [C: 032] New release (v2) of ZppixBot [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370679 (owner: 10Reception123) [19:27:03] (03Merged) 10jenkins-bot: New release (v2) of ZppixBot [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370679 (owner: 10Reception123) [19:37:12] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Patch-For-Review: Build VPS base images - https://phabricator.wikimedia.org/T170828#3510309 (10Andrew) 05Open>03Resolved [19:38:43] 10cloud-services-team (Kanban), 10Patch-For-Review: nova-compute process check flapping - https://phabricator.wikimedia.org/T171606#3510310 (10Andrew) 05Open>03Resolved [19:49:55] (03Draft2) 10Zppix: Update toolinfo.json [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370686 [19:49:59] (03CR) 10Zppix: [C: 032] Update toolinfo.json [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370686 (owner: 10Zppix) [19:50:08] (03Merged) 10jenkins-bot: Update toolinfo.json [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/370686 (owner: 10Zppix) [20:41:34] How do I run a script within a virtual environment from the crontab on toolforge? [20:42:01] I followed https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Using_virtualenv_with_webservice_shell [20:42:43] I have this which should be ok: 20 * * * * /usr/bin/jsub -N cron-tools.wmit-1 -once -quiet /data/project/wmit/www/python/venv/bin/python /data/project/wmit/navierstokes/NavierStokes.py -r 1 [20:43:18] There is one missing dependency on the execution hosts (lynx) so that's probably the culprit, but I'm not sure the cron is correct in the first place [20:50:26] Nemo_bis: that should work to run something every hour [20:50:55] Nemo_bis: there should be some information about what's happening in cron-tools.wmit-1.log and .err [21:01:37] valhallasw`cloud: yeah, it's running [21:02:05] Thanks for confirming I was making some sense :) I'll now fix the error reported in logs ;) [21:30:23] 10Tools: Tool "personabot" loads assets from bootstrapcdn, ajax.googleapis.com, and w3.org - https://phabricator.wikimedia.org/T172822#3510583 (10zhuyifei1999) [21:35:00] 10Tools: Tool "ramp" loads assets from google and cloudflare - https://phabricator.wikimedia.org/T172823#3510601 (10zhuyifei1999) [21:35:24] 10Cloud-Services: 'virt1' entry at markmonitor? - https://phabricator.wikimedia.org/T102689#3510629 (10RobH) So in trying to close old tasks, a search of our markmonitor portfolio doesn't show any domain using virt1.wikimedia.org. Also when editing the nameservers for a domain, I don't see it listed as an optio... [21:35:30] 10Cloud-Services: 'virt1' entry at markmonitor? - https://phabricator.wikimedia.org/T102689#3510632 (10RobH) 05Open>03Resolved [21:36:30] 10Tools: Tool "ramp" loads assets from google and cloudflare - https://phabricator.wikimedia.org/T172823#3510637 (10zhuyifei1999) The tool maintainers are J.little & Timathom. I cannot find the Phabricator username of either of them. [21:40:42] 10Tools: Tool "raun" loads fonts and analytics from google - https://phabricator.wikimedia.org/T172825#3510645 (10zhuyifei1999) [21:43:30] !log rcm Tin: Updating packages [21:43:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:44:47] 10Tools: Tool "remarkup2wikitext" loads assets from code.jquery.com and Google Analytics - https://phabricator.wikimedia.org/T172826#3510668 (10zhuyifei1999) [21:46:30] 10Tools: Tool "replacer" loads bootstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172827#3510683 (10zhuyifei1999) [21:48:13] 10Tools: Tool "ricordisamoa" loads fork-me-on-github ribbon from Amazon AWS - https://phabricator.wikimedia.org/T172828#3510698 (10zhuyifei1999) [21:51:05] 10Tools: Tool "seealsology" loads Google Analytics - https://phabricator.wikimedia.org/T172830#3510728 (10zhuyifei1999) [21:51:39] 10Tools: Tool "seealsology" loads Google Analytics - https://phabricator.wikimedia.org/T172830#3510742 (10zhuyifei1999) The tool maintainer is Boogheta. I cannot find their Phabricator username. [21:56:40] 10Tools: Tool "shuaib" loads assets from d.moviemodeapp.com - https://phabricator.wikimedia.org/T172831#3510744 (10zhuyifei1999) [21:57:06] 10Tools: Tool "shuaib" loads assets from d.moviemodeapp.com - https://phabricator.wikimedia.org/T172831#3510758 (10zhuyifei1999) The other maintainer is Tahir_mq. I cannot find their Phabricator username. [22:01:11] 10Tools: Tool "sqid" loads assets from bootstrapcdn, fonts.googleapis.com, and code.jquery.com - https://phabricator.wikimedia.org/T172833#3510779 (10zhuyifei1999) [22:04:59] 10Tools: Tool "styleguide" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172834#3510793 (10zhuyifei1999)