source: sasmodels/doc/guide/gpu_setup.rst @ 8b31efa

core_shell_microgelsmagnetic_modelticket-1257-vesicle-productticket_1156ticket_1265_superballticket_822_more_unit_tests
Last change on this file since 8b31efa was 8b31efa, checked in by pkienzle, 2 years ago

document cuda device selection; fix cuda speed issue

  • Property mode set to 100644
File size: 7.3 KB
RevLine 
[c0d7ab3]1.. _gpu-setup:
2
3********************
4GPU Setup
5********************
6
[59485a4]7SAS model evaluations can run on your graphics card (GPU) or they can run
8on the processor (CPU). In general, calculations performed on the GPU
9will run faster.
10
11
[c0d7ab3]12OpenCL Installation
13*******************
14*Warning! GPU devices do not in general offer the same level of memory
15protection as CPU devices. If your code attempts to write outside allocated
16memory buffers unpredicatable behaviour may result (eg, your video display
17may freeze, or your system may crash, etc). Do not install OpenCL drivers
18without first checking for known issues (eg, some computer manufacturers
19install modified graphics drivers so replacing these may not be a good
20idea!). If in doubt, seek advice from an IT professional before proceeding
21further.*
22
23Check if you have OpenCL already installed
24==========================================
25
26**Windows**
27
28The following instructions are based on
29http://web.engr.oregonstate.edu/~mjb/cs475/DoIHaveOpenCL.pdf
30
31* Go to: Start -> Control Panel -> System & Security -> Administrative Tools
32* Double Click on Computer Managment
33* Click on Device Manager
34* Click open Display Adapters
35* Right-click on available adapter and select Properties
36* Click on Driver
37* Go to Driver Details
38* Scroll down and see if OpenCL is installed (look for OpenCL*.dll files)
39
40**Mac OSX**
41
42For OS X operating systems higher than 10.6 OpenCL is shipped along with
43the system.
44
45However, OpenCL has had a rocky history on Macs. Apple provide a useful
46compatibility table at https://support.apple.com/en-us/HT202823
47
48
49Installation
50============
51
52**Windows**
53
54Depending on the graphic card in your system, drivers
55can be obtained from different sources:
56
57* NVIDIA: https://developer.nvidia.com/opencl
58* AMD: http://developer.amd.com/tools-and-sdks/opencl-zone/
59
60
61**Mac OSX**
62
63N/A
64
65You cannot download OpenCL driver updates for your Mac. They are packaged
66with the normal quarterly OS X updates from Apple.
67
68
69.. note::
70    Intel provides OpenCL drivers for Intel processors at
71    https://software.intel.com/en-us/articles/opencl-drivers
72    These can sometimes make use of special vector instructions across multiple
73    processors, so it is worth installing if the GPU does not support double
74    precision. You can install this driver alongside the GPU driver for NVIDIA
75    or AMD.
76
77
78GPU Selection
79*************
80
[59485a4]81The logic for choosing the compute platform is a little bit complicated.
[c0d7ab3]82If the model has the line *single=False* then it requires double precision.
83If the GPU is single precision only, then it will try running via OpenCL
84on the CPU.  If the OpenCL driver is not available for the CPU then
85it will run as a normal program on the CPU.
86
87For models with a large number of parameters or with a lot of code,
[59485a4]88the GPU may be too small to run the program effectively. In this case, you
89should try simplifying the model, maybe breaking it into several different
90models so that you don't need *IF* statements in your code. If it is still
91too big, you can set *opencl=False* in the model file and the model will
92only run as a normal program on the CPU. This will not usually be necessary.
[c0d7ab3]93
94Device Selection
95================
[8b31efa]96**OpenCL drivers**
97
[c0d7ab3]98If you have multiple GPU devices you can tell the program which device to use.
99By default, the program looks for one GPU and one CPU device from available
100OpenCL platforms. It prefers AMD or NVIDIA drivers for GPU, and
101prefers Intel or Apple drivers for CPU. Both GPU and CPU are included on
102the assumption that CPU is always available and supports double precision.
103
104The device order is important: GPU is checked before CPU on the assumption that
105it will be faster. By examining ~/sasview.log you can see which device
106was used to run the model.
107
[8b31efa]108If you want to use a specific driver and devices, you can run the following
[c0d7ab3]109from the python console::
110
111    import pyopencl as cl
112    cl.create_some_context()
113
114This will provide a menu of different OpenCL drivers available.
115When one is selected, it will say "set PYOPENCL_CTX=..."
[8b31efa]116Use that value as the value of *SAS_OPENCL=driver:device*.
117
118To use the default OpenCL device (rather than CUDA or None),
119set *SAS_OPENCL=opencl*.
120
121In batch queues, you may need to set *XDG_CACHE_HOME=~/.cache* 
122(Linux only) to a different directory, depending on how the filesystem
123is configured.  You should also set *SAS_DLL_PATH* for CPU-only modules.
124
125    -DSAS_MODELPATH=path sets directory containing custom models
126    -DSAS_OPENCL=vendor:device|cuda:device|none sets the target GPU device
127    -DXDG_CACHE_HOME=~/.cache sets the pyopencl cache root (linux only)
128    -DSAS_COMPILER=tinycc|msvc|mingw|unix sets the DLL compiler
129    -DSAS_OPENMP=1 turns on OpenMP for the DLLs
130    -DSAS_DLL_PATH=path sets the path to the compiled modules
131
132
133**CUDA drivers**
134
135If OpenCL drivers are not available on your system, but NVidia CUDA
136drivers are available, then set *SAS_OPENCL=cuda* or
137*SAS_OPENCL=cuda:n* for a particular device number *n*.  If no device
138number is specified, then the CUDA drivers looks for look for
139*CUDA_DEVICE=n* or a file ~/.cuda-device containing n for the device number.
140
141In batch queues, the SLURM command *sbatch --gres=gpu:1 ...* will set
142*CUDA_VISIBLE_DEVICES=n*, which ought to set the correct device
143number for *SAS_OPENCL=cuda*.  If not, then set
144*CUDA_DEVICE=$CUDA_VISIBLE_DEVICES* within the batch script.  You may
145need to set the CUDA cache directory to a folder accessible across the
146cluster with *PYCUDA_CACHE_DIR* (or *PYCUDA_DISABLE_CACHE* to disable
147caching), and you may need to set environment specific compiler flags
148with *PYCUDA_DEFAULT_NVCC_FLAGS*.  You should also set *SAS_DLL_PATH* 
149for CPU-only modules.
150
151**No GPU support**
152
153If you don't want to use OpenCL or CUDA, you can set *SAS_OPENCL=None*
154in your environment settings, and it will only use normal programs.
155
156In batch queues, you may need to set *SAS_DLL_PATH* to a directory
157accessible on the compute node.
158
[c0d7ab3]159
160Device Testing
161==============
162Unfortunately, not all vendors provide working OpenCL implementations
163for their GPU devices.  For example, the HD 6000 Intel GPUs with
164double precision support fail for some of the double precision models.
165
166The SasView user interface provides a Fitting OpenCL Options dialog
167for selecting amongst and testing the available devices.  After a
168few minutes of seeming to freeze, the application will return a list
169of model tests which have passed.  The same tests can be run directly
170from the python console using::
171
172    from sasmodels.model_tests import main as model_tests
173    model_tests("-v", "opencl", "all")
174
175Compiler Selection
176==================
177For models run as normal programs, you may need to specify a compiler.
178This is done using the *SAS_COMPILER* environment variable, and the
179*SAS_OPENMP* environment variable if OpenMP support is available for
180the compiler.
181
[63602b1]182On Windows, set *SAS_COMPILER=tinycc* for the tinycc compiler,
183*SAS_COMPILER=msvc* for the Microsoft Visual C compiler,
184or *SAS_COMPILER=mingw* for the MinGW compiler. If TinyCC is available
[c0d7ab3]185on the python path (it is provided with SasView), that will be the
186default. If you want one of the other compilers, be sure to have it
187available in your *PATH* so we can find it!
188
189On Mac OS/X and Linux, set *SAS_COMPILER=unix* for the compiler.  This
190will use the unix cc command to compile the model, with gcc style
191command line options.  For OS/X you will need to install the Xcode
192package to make the compiler available.
193
194
195*Document History*
196
[8b31efa]197| 2018-10-15 Paul Kienzle
Note: See TracBrowser for help on using the repository browser.