How to fully parallelize a Java8 stream through buffering?

A

Alex R

Guest
I have some code like this:

Stream<Item> stream = listPaths.parallelStream().flatMap(path -> { ... })


I have also added this:

System.setProperty(
"java.util.concurrent.ForkJoinPool.common.parallelism",
String.valueOf(Runtime.getRuntime().availableProcessors() * 4));


Later I call stream.forEach(...)

However, I have found that on a machine with 32 cores, only 5 to 8 cores are utilized.

I believe what is happening is that the code inside flatMap() and the code inside the forEach() suffer of I/O Latency issues for different external resources, and returns data in "fits and starts" -- a bad combination with the "pull" nature of streams.

Is there a simple (idiomatic, not "go write your own 200 lines of code") to wrap the stream into some sort of "stream buffer" that would keep the source stream fully utilized (pulling at max threads) while feeding the forEach()?

Continue reading...
 
Top