In this study, brown algae (Sargassum glaucescens) nanoparticles were prepared by using a planetary ball mill to remove nickel and cobalt. The biosorption reaction in the reactor was studied under different conditions of pH, biosorbent dose, temperature, and retention time. The concentration of heavy metals was investigated after the fluid had passed through the membrane system. Algae nano-biosorbent was prepared using a planetary ball mill; scanning electron microscope and Brunauer–Emmert–Teller tests showed an average diameter of 95.75 nm and specific surface area of 11.25 m2/g, respectively. A maximum biosorption efficiency equal to 93 and 91 % was achieved for nickel and cobalt at pH 6, temperature 35 °C with a retention time of 80 min, and at biosorbent doses of 8 and 4 g/l. The kinetic data fit well by pseudo-first-order model, and equilibrium data of metal ions could be described well with the Langmuir and Dubinin–Radushkevich isotherm models. The calculated thermodynamic parameters showed that metal ion biosorption is feasible, endothermic, and naturally spontaneous.