how to find the dimensions of a rectangle

The tricky part of this question isn't how to calculate the area of an interior rectangle, but which of all the possible interior rectangles has maximum area?

To start with, observe that the box in your image is the same area regardless of how it is slid around horizontally, and if it is slid to the rightmost wall, it allows for an easy parameterization of the problem as follows:

I find it a bit easier to think of this problem, with the fixed box rotated by the offset angle so that the interior box lines up in a standard orientation. Here's a figure (I've changed theta to beta just because I can type it easily on a mac, and also left off the left most wall for reasons that will be clear):

enter image description here

So think of this constructed as follows: Pick a point on the right side of the exterior rectangle (shown here by a small circle), note the distance a from this point to the corner, and construct the largest possible interior with a corner at this point (by extending vertical and horizontal lines to the exterior rectangle). Clearly, then, the largest possible rectangle is one of the rectangles derived from the different values for a, and a is a good parameter for this problem.

So given that, then the area of the interior rectangle is:

          A = (a * (H-a))/(cosß * sinß)  or, A = c * a * (H-a)

where I've folded the constant trig terms into the constant c. We need to maximize this, and to do that the derivative is useful:

          dA/da = c * (H - 2a)

That is, starting at a=0 (ie, the circle in the figure is in the lower corner of the exterior rectangle, resulting in a tall and super skin interior rectangle), then the area of the interior rectangle increases monotonically until a=H/2 , and then the area starts to decrease again.

That is, there are two cases:

1) If, as a increase from 0 to H/2, the far interior corner hits the opposite wall of the exterior, then the largest possible rectangle is when this contact occurs (and you know it's the largest due to the monotonic increase -- ie, the positive value of the derivative). This is your guess at the solution.

2) If the far corner never touches a wall, then the largest interior rectangle will be at a=H/2.

I haven't explicitly solved here for the area of the interior rectangle for each case, since that's a much easier problem than the proof, and anyone who could follow the proof, I assume could easily calculate the areas (and it does take a long time to write these things up).