How to remove this soap dish?

[EmiJ]

 

 

Hello,

I have this awful soap dish in my shower I want to remove. There is a very tiny allen key opening but it doesn’t seem to work or want to budge when I tried to loosen it. Any ideas on how I could remove the soap dish or should I consider painting it?

Appreciate any suggestions!