Traveling to HK on Indian passport from US

Hi,
I'm planning to visit HK as a tourist for 3 days from US on my way to India, I have an Indian passport but my USA H4 visa is not stamped on the passport. Will that be a problem to travel to HK?
I also have an infant with me who has an American passport, does he need any entry permit for visiting HK?