Yes they are bigger, the manual steering box uses a different steering arm for a more favorable ratio since it's not assisted. The stock assist setup uses the external ram and it's mount on the frame, so it's there where the forces are transmitted not via the box.
it's not that the box attachments are weak, the frame itself is weak, notoriously weak in fact. That big hole for access to those 2 mounting plate for the power steering ram outer mount doesn't help either. If you investigae frames of these cars after a frontal collisio you will always see the frame folded up like a pretzel where that big hole is. A.O. Smith Co. did put a flare (dimple) around the hole to give it some more strength but it's still a weak point. Now IF you were to do a jeep box or whatever conversion that eliminates the external ram, you can weld up that section. Another option is to remove the 2 stud plates and replace them with plates that have 2 nuts or threaded sections welded to them, position in place (bolt down) after drilling some holes in the frame from the bottom up and then plug weld it so the 2 "nut plates" are now securely attached to the frame. Use bolts for the ram instead of studs & nuts.