It is not clear to me what you are trying to do, and why.

But here are some thoughts:

1) Using the Resize Tool to resize a staff is the last thing you should try.

There are a lot of layout problems in using Page Scaling, System Scaling and Staff Scaling simultaneously.

Generally a staff resizing only is relevant for cases like piano & solo instrument.

2) Instead of resizing the staff, consider editing the TAB staff, and increase the distance between the TAB staff-lines.

Then, decrease the size of everything.

(Keep It Simple!)

Here's an objective scenario:

Let's say we aim for these outputs:

• A single TAB staff that is nicely laid out for reading as TAB only
• The same TAB staff (at the same size) that is laid out with a small modern staff above it - for reference purposes, similar to an ossia staff.
• As in bullet 2, the same music again, but laid out nicely for reading the modern notation and the TAB staff is smaller for reference purposes.
• And finally, a version that is only modern notation, nicely laid out for reading/performing.

So, I can see that working with document options and TAB line spacing etc is possible, it just seems like a relatively large amount of trial and error work across multiple instances of a document compared to using a scaling approach (also across multiple instances of a document, but much faster).

Another scenario is as you mentioned - creating a score for an accompanist where the soloist's part is much smaller, for reference only.  In this case the accompanist's part would also be in modern notation.

In that case I suggest that you use the Resize Tool (icon: %), and context-click on the staff.

In the context menu, choose Resize Staff…

(in the context menu you can also access Page Scaling and System Scaling)

Since the layout scalings can quickly become a mess, try to Keep It Simple with Page Scaling and System Scaling.

Set e. g. the System Scaling to 100 % (= no System Scaling), and use only Page Scaling.