[00:17:02] 10cloud-services-team (FY2017-18), 10Goal, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3417401 (10bd808) a:03bd808 [00:28:55] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:42:00] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:03:56] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [01:22:00] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [01:23:45] 10Data-Services, 10Tool-Labs-tools-Xtools, 10cloud-services-team (Kanban), 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3417494 (10jrbs) For what it's worth, this is causing errors on XTools' editcounter (and probably wider) for us... [02:05:37] PROBLEM - Puppet staleness on tools-worker-1020 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [02:15:52] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:50:54] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [03:07:58] 10Data-Services, 10Tool-Labs-tools-Xtools, 10cloud-services-team (Kanban), 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3417594 (10bd808) >>! In T170005#3417494, @jrbs wrote: > For what it's worth, this is causing errors on XTools'... [03:38:40] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:46:52] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:58:36] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [04:21:53] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [04:31:57] (03CR) 10Krinkle: [C: 032] Change #wikimedia-labs to #wikimedia-cloud [labs/striker] - 10https://gerrit.wikimedia.org/r/359042 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [04:34:12] (03Merged) 10jenkins-bot: Change #wikimedia-labs to #wikimedia-cloud [labs/striker] - 10https://gerrit.wikimedia.org/r/359042 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [04:34:39] (03CR) 10Krinkle: [C: 031] "Nice!" [labs/striker] - 10https://gerrit.wikimedia.org/r/358506 (https://phabricator.wikimedia.org/T149458) (owner: 10BryanDavis) [05:36:34] 10Cloud-Services, 10cloud-services-team (Kanban), 10Wikimedia-Mailing-lists: Rename labs-admin mailing list to cloud-admin - https://phabricator.wikimedia.org/T167155#3417717 (10bd808) [07:47:53] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:06:07] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:22:52] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [08:23:22] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:36:07] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [09:28:21] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [09:43:40] 10Toolforge: Node.js on gridengine - https://phabricator.wikimedia.org/T166830#3308975 (10valhallasw) > FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory You likely need to request a larger amount of memory using the `-mem` parameter to `jstart`. [09:43:53] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:27:23] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:02:23] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:18:11] 10Tool-Labs-tools-Xtools: IRC channel name and link update request - https://phabricator.wikimedia.org/T170047#3417960 (10MarcoAurelio) [11:31:51] 10Tool-Labs-tools-Xtools: IRC channel name and link update request - https://phabricator.wikimedia.org/T170047#3417974 (10MarcoAurelio) a:03MarcoAurelio Will give it a try. [11:38:18] 10Tool-Labs-tools-Xtools: IRC channel name and link update request - https://phabricator.wikimedia.org/T170047#3417976 (10MarcoAurelio) https://github.com/x-tools/xtools/pull/129 [11:55:13] 10Tool-Labs-tools-Xtools: IRC channel name and link update request - https://phabricator.wikimedia.org/T170047#3417981 (10MarcoAurelio) ``` diff --git a/public_html/templates/main.php b/public_html/templates/main.php index be3957b..cbb46d2 100644 --- a/public_html/templates/main.php +++ b/public_html/templates/m... [11:56:48] 10Tool-Labs-tools-Xtools, 10User-MarcoAurelio: IRC channel name and link update request - https://phabricator.wikimedia.org/T170047#3417982 (10MarcoAurelio) [12:53:52] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [13:53:24] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:27:28] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:49:58] The real problem with the labs db's, is mariadb isn't smart enough to do constant folding in IF() during optimization [14:50:14] I wonder how hard it would be to make it do that [14:55:27] 10Tool-Labs-tools-Xtools: Automated edits analyzer only works for enwiki - https://phabricator.wikimedia.org/T170050#3418029 (10Framawiki) [14:58:25] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:26] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [15:14:52] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:54:54] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:25] (03CR) 10Paladox: "@Dzahn per @Thcipriani comment at https://gerrit.wikimedia.org/r/#/c/363726/ both private and pub keys need to be added here. The keys can" [labs/private] - 10https://gerrit.wikimedia.org/r/363755 (owner: 10Paladox) [17:22:50] PROBLEM - Puppet errors on tools-exec-1425 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:45:53] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:57:51] RECOVERY - Puppet errors on tools-exec-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:06] 10Tool-Labs-tools-Xtools: Automated edits analyzer only works for enwiki - https://phabricator.wikimedia.org/T170050#3418175 (10MusikAnimal) Checking for tags is a fine idea, but won't account for all tools. In the long run, my plan is to have the regexes separated on a per-project basis, where users can request... [18:20:52] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [18:21:53] 10Tool-Labs-tools-Xtools, 10User-MarcoAurelio: IRC channel name and link update request - https://phabricator.wikimedia.org/T170047#3418190 (10MusikAnimal) 05Open>03Resolved Merged. We actually have a dedicated XTools channel now at wikimedia-xtools, but no need to make another PR, this old version of XToo... [18:36:07] !log tools.zppixbot restarting to clear up some of the bot's cache [18:36:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [18:42:22] 10Data-Services, 10Tool-Labs-tools-Xtools, 10cloud-services-team (Kanban), 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3418219 (10MusikAnimal) >>! In T170005#3417594, @bd808 wrote: >>>! In T170005#3417494, @jrbs wrote: >> For what... [18:51:57] Cyberpower678 i love the iabot interface :) good work mate [19:15:55] Zppix: thanks [19:24:00] Cyberpower678 no problem also i do have one complaint (i created a task for it on phab) [20:12:27] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:57:28] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [21:46:54] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:08:44] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: `maintain-meta_p --all-databases` timeout on labsdb1009 contacting uk.wikimedia.org - https://phabricator.wikimedia.org/T168436#3418296 (10bd808) Led to {T169488} and {T170005}. Instructions at https://wikitech.wikimedia.or... [22:14:30] 10Data-Services, 10Tool-Labs-tools-Xtools, 10cloud-services-team (Kanban), 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3418302 (10bd808) >>! In T170005#3418219, @MusikAnimal wrote: > So for the global edits section of the edit cou... [22:16:39] 10Data-Services, 10Tool-Labs-tools-Xtools, 10cloud-services-team (Kanban), 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3418304 (10bd808) https://tools.wmflabs.org/xtools-ec/?user=Fox&project=en.wikipedia.org&uselang=en works again [22:21:55] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [23:59:41] 10Toolforge: Node.js on gridengine - https://phabricator.wikimedia.org/T166830#3418317 (10edsu) I actually have tried upping the memory. It just takes longer to run out of memory. I would like to upgrade Node if possible.