If you are instead referring to the "rectangular selection frame" drawn to:
* Select tiles in the SBuilderX Imagery Tile Server download dialog from the Zoom / LOD grid tile mosaic ?
* Selecting downloaded tiles within SBuilderX' work-space to be compiled ...from the Background Map tile mosaic ?
If so, AFAIK, those "rectangular selection frames" must always be drawn via SBuilderX' GUI as 'squares' /rectangles.
If, however, you are wanting to only allow portions of your custom photo-real imagery BGL to be displayed in any shape which is effectively NOT a perfect 'square' /rectangle, this done via use of the Blend Mask.
As stated in the tutorial by Luis Feliz-Tirado:
Make photo-real ground textures
It is very easy to create your very own high-resolution, custom (photo-real) ground textures. This document explains the concepts and techniques and illustrates the use of SBuilderX with which you can quickly and easily download aerial images and make this type of scenery. So, why hesitate? Make Flight Simulator scenery as real as it gets! Very sorry - no support of any kind is offered. Please do not write. For any questions, please post in the Avsim Scenery Design Forum.
Filename: make_photo-real_ground_textures_in_fs_x.zip License: Freeware Added: 21st November 2009, 23:20:06 Downloads: 17714 Author: Luis Feliz-Tirado Size: 2143kb
This is new to FS X - it will gradually fade out the custom texture so as to give better transitions to the default ground. The Blend Mask is a gray-scale image, where pure white displays the custom textures and pure black is full transparency (and will display the underlying ground - or water!) All gray values in between will represent varying levels of transparency where the custom textures blend with the default ground.
You can use the Blend Mask to transition between custom land textures and default ground, or you can also use it to fade from your custom water to the default water. "
NOTE: If, rather than 'default' ground, one has OrbX FTX Global land class directly underlying ones own custom (photo-real) ground textures, those OrbX FTX Global land class textures will show instead of "default" land class textures.
PS: A screenshot of what you are endeavoring to achieve would help eliminate 'guessing' here by would-be helpers.