As nobody else has replied, I hope you don't mind if I give it a try.
Your problem does not seem completely clear to me, sorry. And since I am on a dial-up modem, I have not taken a look at your screenshot, too large for me. Nonetheless, let's see:
1. You say that "What surprised me was when I show the underlying mask to add autogen to the fields on the edge of the field the mask isn't square."
Not quite sure what you mean here. The source of the custom texture is rectangular of course, and so is the Blend Mask that you use with it. There is no way around that. Perhaps you could be more specific as to what you mean.
2. You also say "To confuse me more, when I load the bgl to add autogen on to the underlying photoscenery I can only get it to show up to a point, its almost as if the photoscenery created in Sbuilder covers a far greater area than I thought it did."
If you have created the custom texture using a Blend Mask, then some part of the result will be transparent and it will be impossible to add autogen annotations to that. The best thing to do is to first create the custom texture without a Blend Mask, generate the annotations with that, then discard that custom texture and create it again with the Blend Mask. The autogen annotations will then work with the new custom textures and cover all of it.
3. And finally, "even though you can see the boundary of the photoscenery created in Sbuilder, it is still stopping scenery appearing, if I remove it the areas in red appear."
Sorry, again, I cannot see your large screenshot, but there are two things here - you say that scenery does not appear. Just what do you mean by scenery - autogen objects or the custom texture?
Also, the problem of red areas appearing in the custom texture is due to the use of a Blend Mask with a Null Value. This was discussed in a document concerning creation of custom textures:
Make photo-real ground textures in Flight Simulator X
that can be found in the Avsim library.
Basically, if you use a Blend Mask with a Null Value in the inf file, then red spots may appear in the result. The solution is to either remove the Null Value flag from the inf file and generate the custom texture manually, or to select all pure white areas in the source image and make them a color that is not pure white (just darkening them slightly will be fine.)
More details and a more specific explanation concerning your problem would be appreciated.