They have returned but not to a perfect world. They can worship at the site of the destroyed temple but they are still in fear of the neighboring people. Rebuilding the temple meets with some resistance. After reading some of the history of Judah, King Artaxerxes concludes that they can be be dangerous so he forbids them to rebuild the temple.
In our own times, people in a community have felt threatened by the prospect of people of some other religion building a house of worship.