CLUSTER MULTI THREADED NODEJS -- Run threads in parallel to speed up long processes

Publish Date: Aug 3 '20

50 5

My JavaScript that creates thumbnails for hundreds of files is taking too long. So I have decided to speed things up with cluster.fork() that is built into Node.js. This will allow the workload to be spread out across my multi-core processor system.

⚠ 8:53 should read: index % forks === cluster.worker.id - 1

To follow along, you can fork this repository. I have added a tag you can check out that will bring you right to the point this video was created.

git checkout https://github.com/joelnet/mojo-gallery.git
cd mojo-gallery
git checkout youtube-video-cluster

Be sure to subscribe for more videos like this!

Comments 5 total

Mumin GaziAug 3, 2020
5 at most
OMKAR AGRAWALAug 3, 2020
I had a similar requirement, for image manipulation, I used workerpool library. It made task pretty easy. Initially I did achieve it with worker_threads , but the issue I faced was that I couldn't actually keep a track of threads, and would quickly run out of cpus, but workerpool managed it automatically.
- JavaScript JoelAug 3, 2020
  Ahh! this is a cool library. I haven't seen this one before. Definitely helps with the managing of threads. I'll check this out next time I need something multi-threaded. Thanks for sharing!
Jorge RamónAug 5, 2020
What about using Working Threads? Wouldn't be lighter and faster?
- JavaScript JoelAug 17, 2020
  I think you are right. I am familiar with cluster because I use with with Express.js. And I think cluster might be the better use-case when using Express. But in this instance because there isn't much communication back and forth between processes, worker_threads would probably run faster. I think the difference would be small though.
  
  Want to convert it and run some benchmarks? 😁

Add comment

JavaScript Joel @joelnet

CLUSTER MULTI THREADED NODEJS -- Run threads in parallel to speed up long processes

Comments 5 total