I initially thought that it was caused due to rendering that many bullets, but when I ran the profiler, I saw that it was caused by a fault in collision detection. I tweaked it a little, and now it has a better performance.
