I bought a mortising kit for my drill press so I can use square drill bits.

This needs to go onto the quill of the drill press but I can seem to take this plastic part off that the depth stop is attached to.

I've taken the screw out of it but it doesn't seem to want to open up over the chuck.
Any ideas how to remove it? Do I have to remove the quill from the machine?