Do they sell butter or margarine in the larger super markets like Walmart in Beijing.

Yes, of course. Large ones like Walmart have many kind of butter or margarine. Some of these products are import food. You can go and may find the one you often to eat. Different products have different packages. You should choose by yourself.
Butter sounds like "Huang You" (黄油) in mandarin. And margarine sounds like "Ren Zao Nai You" （人造奶油）.