Fire gems can be dropped by Imps and by Saen' Liz. I believe there is also a non hidden chest containing one on B1F.
