When drilling with a masonry bit, you should be making slow progress even if the drill is not in hammer mode. You should be able to drill into it without too much effort.
I propose practicing your drilling technique into a spare brick. Going too slow will make the masonry bit travel, going too fast will heat up the head and ruin the bit.
I suggest using a smaller drill bit to begin with so that you have full control of the masonry bit. Drill to the depth that you need to go to. It is easier keep the drill bit stationary when the bit is not so big. Once you've drilled the primary hole you can then make it bigger using a larger masonry bit.
However, if you still have issues, I suggest upgrading your drill bit to the next level up. - EricL
If you need advice on purchasing a tool, you could go for an all-in-one Ryobi 18V ONE+ HP Brushless Compact Hammer Drill, which is lightweight and capable of drilling holes in masonry whilst also being a great general-purpose drill for screws and holes in timber and steel.
If you already have a non-hammer cordless drill driver, I recommend picking up a budget-friendly 240V hammer drill like the Ozito 710W Hammer Drill. It will be more proficient at drilling holes in masonry than a cordless hammer-function drill driver, but it is a heavier unit. It will drill holes in masonry around twice as fast as a battery-powered unit, so it would be the way to go if you have quite a few holes to drill in a row; if it's just a hole here and there every so often, I think the Ryobi compact hammer drill is a good option.
Remember your PPE when drilling holes in masonry, including a face mask and safety glasses. Also, be mindful of where you are drilling and ensure no electrical cables or pipes run through the wall.
If you are using an SDS rotary drill, one thing to confirm is that you are running your SDS rotary drill in masonry hammer drill mode. Typically, there will be three options, a straight drilling mode for timber and steel, a chiselling mode for removing things like tiles and the masonry hammer drill mode.
Also ensure your masonry drill bits fit into your SDS drill. SDS bits have a specific locking shank. See below for an image of what they look like.
You can try beginning your hole in non-hammer drilling mode to stop your drill from skipping across the surface. Place your drill bit in the right location, and with pressure on the drill, start drilling slowly. Do not go full speed. This will grind away the surface and create an indentation. You can then switch into hammer drill mode and create the hole. The vibrations from the hammer drill mode are what is causing the drill to skip across the surface, and having a locating indentation should assist. Do not run the masonry bit in non-hammer drill mode at high speed, as you'll burn out the tip.
Masonry drill bits can burn out and round over their tips. I've had many customers return to the store with masonry bits not drilling. In most cases, it's from running them too quickly or hitting reo-bar in concrete. If your bit has dulled, it could make it hard to start your hole. - MitchellMc
I always drill a small pilot hole in masonry first. This not only guides the larger bit in, but also leaves a smaller hole to repair should it turn out to be an unsuitable spot for the hole.
I also suggest drilling through a mortar layer without the hammer on, so that it goes in straight and cleanly. Then switch to hammer for the masonry layer. If the hole is a large diameter, you may need to open up the guide hole a little to not overload the large drill bit's cutting capacity. - TedBear
You shouldn't ever need to put a lot of force on a masonry drill for it to drill correctly. The idea is that gentle pressure on the drill unit itself will allow the tip to bore into the material and the flutes in the drill bit to clear the material in the hole as it is drilled.
Putting too much pressure on a masonry bit will ruin the temper (by overheating it) of the carbide tip and make this blunt, rendering the whole drill bit useless, it is that tip that does all of the actual drilling. SDS bits are not cheap and ruining one by mis-use is simply poor economy.
Also, most SDS drill motors/units have a variable speed trigger, if you start off slow, so that the drill bit doesn't jump all over the place, sometimes "pulsing" the trigger gently, the tip will eventually sort itself out and start to bore into the material, just take it quietly until you know that the drill is about 5-10 mm into the hole.
I use a masonry drill quite frequently as an Electrician and I offer the following advice: Let the drill tip do the work.
Be aware that there may be steel reinforcing in the material you are drilling into, this can potentially destroy your drill bit tip, if the bit starts to bend, stop drilling and reverse it out and check the tip, the sides of the tip should look like the picture of the new tip that Mitchell posted above.
You could possibly drill out the reinforcing with a HSS drill bit and some cutting oil, this is very risky and can cause the drill bit to snap in the hole when it contacts the rebar.
Don't ever try to cool a masonry drill bit with water as it is drilling, this is not how they work, they need speed and temperature to cut through the masonry, you can (if you're drilling a large enough hole) end up with steam burns and a wrecked tip at the same time.
What I have done in the past, when I have struck steel reinforcing members in a wall, is get a HSS twist drill bit of the same size hole and coat the end of it with cutting oil (Something like Rocol RTD) and drill out the rebar (remembering to take the "hammer" function of the drill off before doing this).
It will be hard work, do not force the drill bit into the hole, but you will get through it if you keep the tip of the drill wet with cutting oil. Also bear in mind that when the rebar is broken, the bit may catch on the ends of the pieces of steel, so be prepared for that.
A lot of people never even expect to strike reinforcing in a wall inside a house when drilling into a concrete wall.
Chances are, no matter where you drill, 40-50% of the time (if the wall has decent reinforcing in it) you will strike it.
The first sign of this is when your drill bit no longer penetrates the wall, with light pressure on the drill unit itself (which you should always use anyway), it will just stop and go no deeper.
That is why before drilling into any concrete/masonry wall you wrap the shank of the drill bit with 1 or two layers of PVC insulation tape, near the chuck of the drill, this gives you a visual reference of how the drill-bit is travelling into the material.
Should this stop, back the drill bit out, drilling any further is going to destroy the tip on the masonry drill, when it heats up against the steel and will often start to squeal, that is too far and the tip is then damaged.
Tungsten carbide tips on masonry bits are only made for cement and brick, not steel. They are strong, yet brittle and will destroy themselves when over-heated. - MikeTNZ