Load bearing structures rely on the compressive strength of the material being used. Most early structures were, or in some cases still are, load bearing ones. Even the pyramids at Giza are load bearing. The stones at the bottom support all the weight above. Of course there are many stones that contribute to that support. The design of the pyramid (using less and less the higher it gets) is why it is still standing. Those structures that do not use that much material were height restricted. It would require the development of buttresses before a wall could get higher. Of course buttressing is also load bearing as well.