How to move garden bed away from wall

[BigBuilder]

We've recently moved into a home where the garden bed has been built right up against the wall, meaning dirt is sitting directly against the brick possibly damaging it. What is the best way to fix this? Should we dig out the dirt and add a backing to the garden bed? (This could be complex as the decking would need to adjusted and the soil is solid clay). Or should we cut out loss and remove the whole thing and come up with a different solution? 

PS ignore the bamboo, another little gift from the previous owner 😆