Say you want to buy two buildings next to each other and replace them with a bigger building. Suddenly you need two people to sell, not just one. And because only a small fraction of buildings are on the market at any given time it's unlikely to find two sellers adjacent to each other.
Sure, you can ask people to sell. And sometimes offering more money fixes the problem. And sometimes it doesn't. "This is my home" is a hard emotion to overcome.
For more on this topic, see how this affects Los Angeles.
As always, there is another option, like accepting the constraints of existing lots. See this impressive development in San Francisco for an example.