is it just that it needs time or it's like a bug?
No, it just takes time. The second wall is orders of magnitude more resistant than the first one, but after 1 hour of investing in Break alchemy, balloons and fire damage, it's down to about 90% health.
