When running `calculate_doublet_performance` or `calculate_doublet_performance_stochastic` (described in more detail in [deterministic doublet](deterministic_doublet.md) and [stochastic_doublet](stochastic_doublet.md)) then each combination of input reservoir properties is an independent simulation.
When running `calculate_doublet_performance` or `calculate_doublet_performance_stochastic` (described in more detail in [deterministic doublet](deterministic_doublet.md) and [stochastic_doublet](stochastic_doublet.md)) then each combination of input reservoir properties is an independent simulation, making this computation a good target for parallelization, where you use more hardware resources to run processes simultaneously to decrease the execution time.
This makes this a good target for parallelization, where you use more hardware resources to run processes simultaneously to decrease the execution time.
Traditionally trying to parallelize code in python has been tricky and modules such as [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) have been developed to handle this task, still a lot of custom code usually has to be developed to get the right setup for your specific problem.
Traditionally trying to parallelize code in python has been tricky and custom in built modules such as [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) have been developed to handle this task, still a lot of custom code usually had to be developed to get the right setup for your specific problem.
pythermogis however uses [xarray](https://docs.xarray.dev/en/latest/index.html) which under the hood uses [dask](https://www.dask.org/) to run paralel operations. For more details on how xarray utilizes dask for easy parallelization we direct the reader to the following: [Parallel Computing with Dask](https://docs.xarray.dev/en/latest/user-guide/dask.html).
pythermogis however uses [xarray](https://docs.xarray.dev/en/latest/index.html) as its data framework which under the hood uses [dask](https://www.dask.org/) to run parallel operations. For more details on how xarray utilizes dask for easy parallelization we direct the reader to the following: [Parallel Computing with Dask](https://docs.xarray.dev/en/latest/user-guide/dask.html).
The framework has already been implemented in pythermogis, the user simply has to define the `chunk_size` parameter when calling either `calculate_doublet_performance` or `calculate_doublet_performance_stochastic` and then the doublet simulations will occur in parallel.
See below for an explaination of what `chunk_size` is and how to determine the optimal size.
## What is chunk size and how to determine it?
dask parallelization works by applying an operation (in this case `simulate_doublet`) across 'chunks' of input data, which are a collection of data that are run in parralel.
## What is chunk size and how to determine the optimal value?
dask parallelization works by applying an operation (in this case `simulate_doublet`) across 'chunks' of input data in parallel.
Lets say we wish to compute 1000 doublet simulations our smallest `chunk_size` would be 1, meaning that every simulation is sent as an independent job to a processor, while the largest `chunk_size` is 1000, meaning one job is sent to a processor and the simulations are run in series.
As an example, say we wish to compute 1000 doublet simulations our smallest possible `chunk_size` would be 1, meaning that every simulation is sent as an independent job to a processor, while the largest `chunk_size` is 1000, meaning one job is sent to a processor and the simulations are run in series.
The first example would be inefficient as there is a computational cost to chunking when it comes to organising the input and the output, while the second example is also inefficient as each simulation is run in series, the optimal `chunk_size`is likely to be between these two values.
The first example would be inefficient as there is a computational cost to chunking and de-chunking the input and the output, while the second example is also inefficient as each simulation is run in series, the optimal `chunk_size`will be between these two values.
The following figure shows how different chunk sizes affects the overall compute time. It can be seen that the most efficient chunk size (for the hardware this example was run on) is by having 100 simulations per chunk.
The following figure shows how different chunk sizes affects the overall compute time. It can be seen that the most efficient chunk size (for the hardware this example was run on) is by having 100-200 simulations per chunk.
To aid in assessing the optimal chunk size and to make the above figure pythermogis has a function you can run on your hardware: `assess_optimal_chunk_size` (see [assess optimal chunk size](../reference/assess_optimal_chunk_size.md) for usage).
This runs the same set of doublet simulations using different chunk sizes and prints the results to the terminal to show find which chunk size is optimal.
It runs each chunk size 3 times and takes the average of their times to find the time taken for that chunk size.
# generate simulation samples across desired reservoir properties
print(f"parralel simulation, chunk size: {sample_chunk}, took {np.mean(time_attempt):.1f} seconds to run {n_simulations} simulations, {n_simulations/mean_time[-1]:.1f} samples per second")
