[00:31:53] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:48:49] !log git installed mediawiki vagrant for ldap [00:48:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [00:49:07] !log git switch ldap on gerrit to gerrit-test3.git.eqiad.wmflabs [00:49:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [01:06:56] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:50] 10Tools: Requesting access to tools.speedydeletionwikia for Dylann1024 (Nathan Larson) - https://phabricator.wikimedia.org/T171130#3459123 (10Reedy) [02:15:11] 10Tools: Requesting access to tools.speedydeletionwikia for Dylann1024 (Nathan Larson) - https://phabricator.wikimedia.org/T171130#3454957 (10Reedy) If you were filing this request, because you couldn't add them yourselves, you should have mentioned this originally [03:21:00] 10cloud-services-team (FY2017-18), 10Goal: Program 4 Outcome 1: improve documentation - https://phabricator.wikimedia.org/T166401#3459179 (10bd808) Almost all of the Cloud Services outcomes are multi-quarter or perpetual projects, so I made tracking tasks for each of them and then expect to tie other more acti... [03:42:59] 10VPS-project-XTools, 10Community-Tech-Sprint: Internal Server Error from new articleinfo interface in XTools - https://phabricator.wikimedia.org/T169767#3459209 (10Samwilson) 05Open>03Resolved Yep, all is set up correctly now. The remaining other errors will be fixed separately. (@MusikAnimal you're happ... [03:50:45] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:01:08] 10VPS-project-XTools, 10Collaboration-Team-Triage, 10Community-Tech, 10Flow: Add Flow contributions to Xtools - https://phabricator.wikimedia.org/T136950#3459212 (10Samwilson) a:05Samwilson>03None [04:07:16] 10VPS-project-XTools: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3459215 (10Samwilson) [04:08:01] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3459228 (10Samwilson) p:05Triage>03Normal [04:12:36] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3459233 (10Samwilson) [04:20:44] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [04:44:32] 10VPS-project-XTools, 10Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3459242 (10Matthewrbowker) [04:44:34] 10VPS-project-XTools, 10User-Matthewrbowker: Convert all xtools issues to Phabricator - https://phabricator.wikimedia.org/T134632#3459240 (10Matthewrbowker) 05stalled>03Open Reopening, as we are now actively closing issues attached to the xtools-legacy repo. [04:50:21] 10VPS-project-XTools, 10Wikibugs: Update XTools - https://phabricator.wikimedia.org/T171265#3459243 (10Matthewrbowker) [05:05:41] (03PS1) 10Matthewrbowker: Update project name for XTools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) [05:39:04] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3365842 (10Liuxinyu970226) @bd808 shouldn't you rename the logo too? {F8804754} [07:05:06] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3459342 (10Samwilson) The killer is actually killing now; see https://xtools.wmflabs.org/killed_slow_queries.txt for its victims. The problem was that it was querying the wrong database ser... [07:26:41] 10Cloud-Services, 10Toolforge, 10DBA: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3459377 (10Marostegui) [07:26:45] 10Cloud-Services, 10Toolforge, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#3459374 (10Marostegui) 05Open>03Resolved a:03kaldari Thanks @kaldari, this went down to 35G ``` # du -sh /srv/sqldata/p50380g50816__pop_stats/ 35G /... [08:40:15] ugh, wikitech api doesn't seem to return a list of instances anymore [08:40:17] https://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niregion=eqiad&format=json&niproject=deployment-prep [08:42:52] 10VPS-project-XTools: XTools Edit Counter does not report admin actions - https://phabricator.wikimedia.org/T171278#3459521 (10Peachey88) [08:43:05] 10VPS-project-XTools: XTools Edit Counter does not report year counts or month counts - https://phabricator.wikimedia.org/T171277#3459522 (10Peachey88) [09:06:19] 10VPS-project-XTools, 10Documentation, 10User-Matthewrbowker: Document algorithm for AdminScore - https://phabricator.wikimedia.org/T170892#3459552 (10Matthewrbowker) https://github.com/x-tools/xtools/pull/57 [09:09:46] 10Cloud-Services, 10Operations: wikitech api action=query not returning list of instances - https://phabricator.wikimedia.org/T171280#3459555 (10fgiunchedi) [09:11:10] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3459570 (10fgiunchedi) [09:19:58] filed as ^ [09:27:19] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:42:59] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3459555 (10hashar) Code is in api/ApiListNovaInstances.php. Replaying it on silver: ``` $ mwscript eval.php --wiki=labswiki > global $wgOpenStackManagerLDAPUsername; > glo... [09:46:43] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3459705 (10hashar) And in the nova logs, I also see 401 for the tools project for requests from Silver "GET /v2/tools/servers/detail HTTP/1.1" status: 401 len: 291 [09:54:50] 10VPS-project-XTools: XTools Edit Counter does not report year counts or month counts - https://phabricator.wikimedia.org/T171277#3459722 (10Aklapper) @Hawkeye7: Thanks for reporting! For future reference, please associate a project tag when possible. Thanks. [10:02:19] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [10:03:02] 10PAWS: "404 - Not found" when I tried to access to PAWS Control Panel - https://phabricator.wikimedia.org/T140525#2467657 (10Aklapper) Cannot reproduce. I am at https://paws.wmflabs.org/paws/user/MyUserName/tree? and click "Control Panel" at the top and end up on https://paws.wmflabs.org/paws/hub/home [10:03:36] 10PAWS: HTTP 404 error (or sometimes 502 error) when trying to access the Control Panel of PAWS - https://phabricator.wikimedia.org/T140525#3459746 (10Aklapper) [10:07:47] 10PAWS, 10Jupyter-Hub: HTTP 502 Bad Gateway error when trying to log into my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#3459768 (10Aklapper) [10:21:43] 10Cloud-Services, 10Operations, 10Security: labspuppetmaster security issues - https://phabricator.wikimedia.org/T171289#3459835 (10faidon) [10:22:22] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:44:33] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3459889 (10fgiunchedi) There's also a related alert for `novaadmin has roles in every project` which I believe it is related, asking for instances in a project not listed... [10:57:22] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:33] 10Tools, 10Wikidata: Small-displayed images false positive at wp_no_image - https://phabricator.wikimedia.org/T171033#3460165 (10Magnus) I wrote the tool. It uses the "page image" information from the respective Wikipedia. For your example, there is no such entry. Ergo, this needs to be fixed in MediaWiki core... [12:21:29] I'm currently unable to log into some (not all) of my labs instances. Would someone be able to take a look for me? I can happily log into puppetmaster-01.wikifactmine.eqiad.wmflabs but not into elasticsearch-01.wikifactmine.eqiad.wmflabs [12:22:06] I get public key denied. Notable is that hosts I can't log into are using my own puppetmaster [12:26:42] 10PAWS: HTTP 404 error (or sometimes 502 error) when trying to access the Control Panel of PAWS - https://phabricator.wikimedia.org/T140525#3460212 (10Strainu) @Aklapper, have you tried doing so with a bot account while having a server started on your main account? The bot probably has to have MainAccountBot as... [12:28:27] tarrow hi, could you restart those instances? [12:28:45] reason why is we need to see if puppet ran there but it needs a service to be restarted [12:28:51] (ldap needs a new cert) [12:30:37] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:32:18] I've restarted one instance but it made no difference. I would guess I need to actually pull the latest role to my puppetmaster then [12:32:56] tarrow oh, do you have to manually update your puppet master? [13:10:35] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [13:34:03] it seems so; I seem to need to periodically git pull. I guess eventually the auto update process fails to run [13:52:24] This doesn't seem to fix the problem. Is there a ticket explaining the LDAP problems a few days ago. I guess it is related? [13:54:55] tarrow you wont be able to ssh in if you have to manually update the puppetmaster. [13:55:12] Someone from the cloud team would have to ssh in and git pull it :) [14:06:49] tarrow: I can look, just give me a minute [14:09:26] tarrow: you have a local change on your puppetmaster which is preventing updates I think. I'll put it in a patch... [14:12:47] tarrow: better now? [14:18:07] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3460474 (10hashar) ``` lang=json $ curl 'https://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niregion=eqiad&format=json&niproject=deployment-prep' | j... [14:26:35] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3459555 (10Andrew) There was a brief period when novaadmin couldn't log in, is it possible you just caught it at a bad moment? The above curl seems ok to me now. [14:30:36] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3460516 (10hashar) Yup because I have added `novaadmin` as a member of the `deployment-prep` tenant. But for `tools` it is still empty: ``` $ curl 'https://wikitech.wikim... [14:38:21] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3460543 (10fgiunchedi) >>! In T171280#3460500, @Andrew wrote: > There was a brief period when novaadmin couldn't log in, is it possible you just caught it at a bad moment... [14:49:12] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3460618 (10Andrew) I just can't think of any reason why those roles would've been removed :( investigating [15:03:26] andrewbogott: The good news is I can ssh to the puppetmaster (it wasn't master of its self); the better news is you fixed my problem. What did you have to do? [15:03:52] tarrow: the puppet repo on your puppetmaster had a local change that blocked updates. [15:03:58] So I put that in a patch and did a fetch and rebase [15:04:04] and then ran puppet on the client [15:04:20] ah, I thought I'd fixed that. Guess I hadn't [15:05:08] Is the right thing to do just commit my local changes and then merge upstream or do I need to do something more? [15:05:39] you can keep a local change local if it's in a patch [15:05:43] it's just actual file-diffs that upset git [15:05:52] I don't know what you mean by 'merge upstream' [15:06:01] in theory the puppet repo will automatically fetch and rebase if it's able [15:08:20] andrewbogott: how hard is it to learn puppet? [15:08:45] puppet is annoying but no worse than any other language, just different [15:15:01] Like, I want to right instructions for re-creating my current data store, and it should be pretty simple: redis, a handful of python libraries, etc. [15:15:10] s/right/write [15:16:36] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3460744 (10bd808) >>! In T168480#3459278, @Liuxinyu970226 wrote: > @bd808 shouldn't you rename the logo and texts on https://tools.wmflabs.org/... [15:17:48] 10VPS-project-XTools, 10Community-Tech-Sprint: Internal Server Error from new articleinfo interface in XTools - https://phabricator.wikimedia.org/T169767#3460745 (10MusikAnimal) Yup! Hopefully ArticleInfo's legacy of internal server errors are coming to a close :) [15:18:13] andrewbogott: What do you mean by patch? A git commit? Or is there some way of storing puppet patches outside of that kept in git? [15:18:22] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:18:29] yes, a commit [15:23:20] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:32:24] 10VPS-project-XTools: XTools Edit Counter does not report admin actions of former admins - https://phabricator.wikimedia.org/T171278#3460776 (10MusikAnimal) [15:35:07] 10VPS-project-XTools: XTools Edit Counter does not report totals for each year and month - https://phabricator.wikimedia.org/T171277#3460779 (10MusikAnimal) [15:41:12] 10VPS-project-XTools: XTools Edit Counter does not report totals for each year and month - https://phabricator.wikimedia.org/T171277#3460823 (10MusikAnimal) I have some Chart.js code that might help with this: https://github.com/MusikAnimal/pageviews/blob/master/javascripts/shared/chart_helpers.js#L856-L902 I t... [15:53:20] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:16] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3460841 (10MusikAnimal) Yay! Thanks for this. Hopefully this will solve the issue @DannyH ran into the other day, where one of the Apache instances was hanging. Just a guess... I have the sa... [15:56:47] 10VPS-project-XTools, 10translatewiki.net: Add translatewiki.net support for XTools - https://phabricator.wikimedia.org/T170789#3460843 (10MusikAnimal) a:03MusikAnimal [16:03:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:51] 10wikitech.wikimedia.org: novaadmin removed from many keystone projects - https://phabricator.wikimedia.org/T171313#3460869 (10Reedy) [16:08:35] 10VPS-project-XTools, 10translatewiki.net: Add translatewiki.net support for XTools - https://phabricator.wikimedia.org/T170789#3460882 (10MusikAnimal) Gerrit patch: https://gerrit.wikimedia.org/r/#/c/366863/ [16:18:50] 10wikitech.wikimedia.org: novaadmin removed from many keystone projects - https://phabricator.wikimedia.org/T171313#3460909 (10Andrew) So currently I think this was caused by a misfire in OpenStackManager's removeUserFromBastionProject(): 2017-07-20T19:22:17 BryanDavis (talk | contribs | block) changed group m... [16:24:22] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:25:52] 10VPS-project-XTools, 10Wikibugs, 10Patch-For-Review: Update XTools on Wikibugs - https://phabricator.wikimedia.org/T171265#3460954 (10MusikAnimal) [16:26:18] (03CR) 10MusikAnimal: [C: 04-1] "Let's hold off on this. I think our project name should just be "XTools". Going to create a ticket" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [16:29:38] 10VPS-project-XTools, 10Wikibugs, 10Patch-For-Review: Update XTools on Wikibugs - https://phabricator.wikimedia.org/T171265#3459243 (10MusikAnimal) Let's hold off on this. I think our project name should just be "XTools". For this I've created T171323 [16:30:25] 10VPS-project-XTools, 10Phabricator: Rename "VPS-project-XTools" to "XTools" - https://phabricator.wikimedia.org/T171323#3460976 (10MusikAnimal) [16:59:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:37] 10Cloud-Services, 10Toolforge, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#3461351 (10Niharika) I don't think we need to keep those 35G worth of data anymore even. The bot is gone forever and that data is pretty outdated. I can't... [17:46:50] 10cloud-services-team (Kanban), 10Project-Admins, 10User-bd808: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244#3461367 (10mmodell) @bd808: Done. One of them didn't work though: ``` The selected child project already has subprojects or milestones of its o... [17:51:57] 10cloud-services-team (Kanban), 10Project-Admins, 10User-bd808: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244#3461391 (10MarcoAurelio) https://phabricator.wikimedia.org/project/subprojects/1821/ both are archived and have no tasks; feel free to delete them... [17:55:20] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:58:54] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3459215 (10kaldari) @Samwilson: Wow, that log is already enormous. Do we have log rotation in place for it? Maybe we should have it only log the killed queries rather than every single check... [18:02:10] (03Draft2) 10MarcoAurelio: Wikimedia Labs rebranding [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/366881 [18:02:15] (03Draft1) 10MarcoAurelio: Wikimedia Labs rebranding [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/366881 [18:23:13] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3461461 (10MusikAnimal) This is apparently killing queries from the legacy XTools, as evidenced by the presence of `s51187__xtools_tmp`. I also noticed it killed a query that processed [[... [18:34:53] 10VPS-project-XTools, 10Community-Tech-Sprint: Long queries not being killed - https://phabricator.wikimedia.org/T171264#3461477 (10MusikAnimal) Another oddity, some queries on databases other than enwiki are crazy slow. Enwiki I assume has the biggest logging table, but check this out: ```lang=sql SELECT log_... [19:00:21] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:23] 10Cloud-Services, 10Toolforge, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#3461652 (10Marostegui) If you guys think it can be dropped...go ahead! :-) [20:14:39] 10cloud-services-team (Kanban), 10Project-Admins, 10User-bd808: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244#3461787 (10mmodell) Even after removing the two subprojects it still refuses. I guess I'll have to do some manual fiddling with the database. [20:15:16] (03CR) 10Matthewrbowker: "Okay. Because it was changed without any discussion (that I saw), I assumed the name was a decision handed down from "on high." I can am" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [20:33:21] 10cloud-services-team (Kanban), 10Project-Admins, 10User-bd808: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244#3461809 (10bd808) Bask in the glory of . We have hierarchy again that was lost in the B... [20:37:01] bd808: Yay. [20:37:14] its a tree! [20:37:30] and possibly a tiny bit more discoverable [20:37:36] * James_F nods. [21:00:05] 10Wikibugs, 10XTools, 10Patch-For-Review: Update XTools on Wikibugs - https://phabricator.wikimedia.org/T171265#3459243 (10Quiddity) project name updated, now ready for updated patch [21:52:59] 10wikitech.wikimedia.org: novaadmin removed from many keystone projects - https://phabricator.wikimedia.org/T171313#3462031 (10Andrew) [21:53:02] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3462030 (10Andrew) [22:12:37] 10cloud-services-team (Kanban), 10Project-Admins, 10User-bd808: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244#3462101 (10bd808) [22:16:12] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3462124 (10Andrew) 05Open>03Resolved a:03Andrew I have a fix to prevent this from happening again... in the meantime I've added novaadmin back to everything. [22:24:51] (03CR) 10BryanDavis: [C: 032] Wikimedia Labs rebranding [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/366881 (owner: 10MarcoAurelio) [22:25:17] (03Merged) 10jenkins-bot: Wikimedia Labs rebranding [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/366881 (owner: 10MarcoAurelio) [22:34:53] (03PS2) 10BryanDavis: Update project name for XTools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [22:36:14] (03CR) 10BryanDavis: [C: 031] Update project name for XTools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [23:04:50] (03CR) 10Legoktm: [C: 032] Update project name for XTools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [23:06:24] (03Merged) 10jenkins-bot: Update project name for XTools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [23:06:31] (03CR) 10jenkins-bot: Update project name for XTools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/366786 (https://phabricator.wikimedia.org/T171265) (owner: 10Matthewrbowker) [23:16:12] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:37:13] 10Cloud-Services, 10cloud-services-team (Kanban), 10Operations, 10Patch-For-Review: Reimage labstore1001 and labstore1002 for DRBD storage setup - https://phabricator.wikimedia.org/T158196#3029409 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by madhuvishy on neodymium.eqiad.wmnet for hosts:... [23:56:11] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0]