Should I apply for regular Visa (Canadian citizen) if I am likely to be rejected (no birth certicate

Birth in HK would confer automatic Chinese citizenship if you are of Chinese ancestry which will make you eligible for an HKSAR passport and an HK ID card. In this case you will not lose Chinese citizenship by naturalizing in another country and some consulates (Toronto being one of them) have refused visas to this group of people on the ground that they are not foreign. Unless your Chinese citizenship is renounced, the lawful way is for you to obtain a Home Return Permit but that requires applying for an HKSAR passport and ID card first. I've never heard of the 20-year rule and some migrants who have lived in Canada for decades have had their visas refused.

Onto whether you are eligible for visa-free transit: if you do not have a visa refusal stamp in your passport then it should be OK to immigration (although the most orthodox way is still to travel with HRP and HKSAR passport). If you do, you need a new passport as immigration will not let you in as long as there is a refusal stamp.