The traditional method of removing rust stains on vinyl flooring is using bi-carb in combination with vinegar. Spread the bi-carb on the stained area and use a clean rag soaked in vinegar and vigorously wipe the area. Use a rag soaked in water to do the final cleanup.
If not successful, the next step up would be to use chemical cleaners. Please test a small area first to make sure it will not damage your vinyl floor.