Parents may not tend to the babies if you are too close, so watch the nest from a distance. If it's not possible for you, wedge a small stick or weed stem in the entrance hole and come back later to see if the nest remains.
Let's Get It Fixed!
HomeX virtual experts will assess your issue (for free!) and can resolve simple problems.