Nowadays, there are many libraries to solve your problem. OpenCV (image processing and computer vision library written in c) has implementation for Hough transform (the simple hough transform find lines in an image, while the generalized one finds more complex objects) so that could be a good start. This transformation is also used to find faces in an image and reflect them with a rectangle or square. For the rectangles which do have closed corners, there are also corner detectors such as cornerHarris which can help.
I ran the hough lines demo provided with OpenCV and here's the result on the image you gave (detected lines marked in red):
In the above image, you can see that wherever the closed rectangular patterns were observed, the algorithm has drawn some specific rectangular shapes there.
Hope this answer helps.