after the quest is finished. Very annoying when looking for slightly more important things in that area.
It's because you keep the fancy plant pot recipe and can make more as house items; you need the clay deposit to make them (they are now essentially a harvestable crafting ingredient).
Type "-clay" into the search bar to filter it out (have not checked, hopefully "clay" isn't part of the name of any other item).