Instead, when these people were released from Babylon, they worried about their own homes, fields, and vineyards. Their works was doubly hard because they faced obstacles to their work, especially famine.
Haggai revealed that the problem was with their behavior. While they worked long hours on their own properties, the former Temple was simply a pile of ruins. As long as God's house remained unbuilt, God would limit the fruits of the people's labors. However, if they return to God in faith -- and rebuilding the Temple is one part of such faith -- God will reward their efforts. Eventually, they will gain the wealth of other nations (after they are punished by God for their sinfulness).