What do you mean by
no part of the covering shape may lie outside both polygons.
and by
A shape that leaves a smaller percentage of its own area uncovered is a better solution.
?
You may not put the covering shape (even partly) outside the triangle or square.
The edges of both polygons may be tangents, but not secants of the covering shape.
Those covering shape has its own area - A. By tiling those two polygons some area could stay uncovered. This whole uncovered area of both polygons together has to be a smallest possible percentage of A. Preferably 0.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "