Without workers you cannot perform work – your population is the key here. Your residential buildings make people available to you, who you will need to build other buildings such as production buildings, goods buildings and military buildings. Make sure to have enough free population, otherwise you cannot build these buildings. If you demolish a residential building, you lose the population that inhabited it. If you do not have enough free population, you cannot tear down your residential building. You can also get back the population: If you sell a building the used population will be freed and again added to your population pool.