If I wear a L (tees) and M (cami's) at Hollister what size would I be for Ralph Lauren?

My problem is that my chest won't fit.. Cause I'm checking the size chart and I'm still deciding because my waist is smaller than the large but my chest is bigger than the medium.. just wanted to know what others would do..