Hi @lenstaa,
The circles you are seeing around the mounting hole are where the plasterboard has fractured and pulled outwards under the weight of the shelf. Unfortunately, once plasterboard has fractured, the gypsum, which makes up the core of the plasterboard, is no longer attached to the rest of the gypsum around it, so there is no longer any strength in this area. In this situation, there is nothing you can really do besides cut it out and replace it or patch it.
To fix this issue, you are going to have to cut around the outer circle, remove the fractured plasterboard and then patch the larger hole. Ensuring you wear gloves and a long sleeve work shirt to limit the risk of lacerations, you can use a utility knife to cut around the fractured plasterboard to widen the hole.
You can then follow How To Patch A Plaster Wall for some guidance on how to patch the enlarged hole.
If you plan to hang something from this area, this solution would not be sufficient and you would have to patch the hole using the technique shown in How To Repair A Large Hole In Plasterboard.
Let me know what you think and if you have further questions, please don't hesitate to ask.
Jacob
