In the presented work we have performed Brownian dynamic simulations to evaluate the diffusion constants of nonmagnetic particles in magnetic fluids. We have observed a strong anisotropy for these diffusion constants with reference to the applied magnetic field. The diffusion constants of these particles have a lower value in the direction perpendicular to the external magnetic field as compared to the direction parallel to the applied magnetic field. This anisotropy was studied as a function of magnetic particle size, magnetic particle concentration and nonmagnetic particle size.