Changeset 8b31efa in sasmodels

Timestamp:

Oct 15, 2018 1:27:14 PM (6 years ago)

Author:

pkienzle

Branches:

master, core_shell_microgels, magnetic_model, ticket-1257-vesicle-product, ticket_1156, ticket_1265_superball, ticket_822_more_unit_tests

Children:

508475a, d5ce7fa

Parents:

4de14584

Message:

document cuda device selection; fix cuda speed issue

Files:

• doc/guide/gpu_setup.rst (modified) (4 diffs)
• sasmodels/kernelcl.py (modified) (2 diffs)
• sasmodels/kernelcuda.py (modified) (4 diffs)

doc/guide/gpu_setup.rst

@@ -94 +94 @@
 Device Selection
 ================
+**OpenCL drivers**
+
 If you have multiple GPU devices you can tell the program which device to use.
 By default, the program looks for one GPU and one CPU device from available
@@ -104 +106 @@
 was used to run the model.
 
-**If you don't want to use OpenCL, you can set** *SAS_OPENCL=None*
-**in your environment settings, and it will only use normal programs.**
-
-If you want to use one of the other devices, you can run the following
+If you want to use a specific driver and devices, you can run the following
 from the python console::
 
@@ -115 +114 @@
 This will provide a menu of different OpenCL drivers available.
 When one is selected, it will say "set PYOPENCL_CTX=..."
-Use that value as the value of *SAS_OPENCL*.
+Use that value as the value of *SAS_OPENCL=driver:device*.
+
+To use the default OpenCL device (rather than CUDA or None),
+set *SAS_OPENCL=opencl*.
+
+In batch queues, you may need to set *XDG_CACHE_HOME=~/.cache* 
+(Linux only) to a different directory, depending on how the filesystem 
+is configured.  You should also set *SAS_DLL_PATH* for CPU-only modules.
+
+    -DSAS_MODELPATH=path sets directory containing custom models
+    -DSAS_OPENCL=vendor:device|cuda:device|none sets the target GPU device
+    -DXDG_CACHE_HOME=~/.cache sets the pyopencl cache root (linux only)
+    -DSAS_COMPILER=tinycc|msvc|mingw|unix sets the DLL compiler
+    -DSAS_OPENMP=1 turns on OpenMP for the DLLs
+    -DSAS_DLL_PATH=path sets the path to the compiled modules
+
+
+**CUDA drivers**
+
+If OpenCL drivers are not available on your system, but NVidia CUDA
+drivers are available, then set *SAS_OPENCL=cuda* or
+*SAS_OPENCL=cuda:n* for a particular device number *n*.  If no device
+number is specified, then the CUDA drivers looks for look for
+*CUDA_DEVICE=n* or a file ~/.cuda-device containing n for the device number.
+
+In batch queues, the SLURM command *sbatch --gres=gpu:1 ...* will set
+*CUDA_VISIBLE_DEVICES=n*, which ought to set the correct device
+number for *SAS_OPENCL=cuda*.  If not, then set
+*CUDA_DEVICE=$CUDA_VISIBLE_DEVICES* within the batch script.  You may
+need to set the CUDA cache directory to a folder accessible across the
+cluster with *PYCUDA_CACHE_DIR* (or *PYCUDA_DISABLE_CACHE* to disable
+caching), and you may need to set environment specific compiler flags
+with *PYCUDA_DEFAULT_NVCC_FLAGS*.  You should also set *SAS_DLL_PATH* 
+for CPU-only modules.
+
+**No GPU support**
+
+If you don't want to use OpenCL or CUDA, you can set *SAS_OPENCL=None*
+in your environment settings, and it will only use normal programs.
+
+In batch queues, you may need to set *SAS_DLL_PATH* to a directory
+accessible on the compute node.
+
 
 Device Testing
@@ -154 +195 @@
 *Document History*
 
-| 2017-09-27 Paul Kienzle
+| 2018-10-15 Paul Kienzle

sasmodels/kernelcl.py

@@ -227 +227 @@
         self.context = None
         if 'SAS_OPENCL' in os.environ:
-            #Setting PYOPENCL_CTX as a SAS_OPENCL to create cl context
-            os.environ["PYOPENCL_CTX"] = os.environ["SAS_OPENCL"]
+            # Set the PyOpenCL environment variable PYOPENCL_CTX 
+            # from SAS_OPENCL=driver:device.  Ignore the generic
+            # SAS_OPENCL=opencl, which is used to select the default 
+            # OpenCL device.  Don't need to check for "none" or
+            # "cuda" since use_opencl() would return False if they
+            # were defined, and we wouldn't get here.
+            dev_str = os.environ["SAS_OPENCL"]
+            if dev_str and dev_str.lower() != "opencl":
+                os.environ["PYOPENCL_CTX"] = dev_str
+
         if 'PYOPENCL_CTX' in os.environ:
             self._create_some_context()
@@ -568 +576 @@
                 current_time = time.clock()
                 if current_time - last_nap > 0.5:
-                    time.sleep(0.05)
+                    time.sleep(0.001)
                     last_nap = current_time
         cl.enqueue_copy(self.queue, self.result, self.result_b)

sasmodels/kernelcuda.py

@@ -444 +444 @@
         self.q_input = q_input # allocated by GpuInput above
 
-        self._need_release = [self.result_b, self.q_input]
+        self._need_release = [self.result_b]
         self.real = (np.float32 if dtype == generate.F32
                      else np.float64 if dtype == generate.F64
@@ -467 +467 @@
         # Call kernel and retrieve results
         last_nap = time.clock()
-        step = 1000000//self.q_input.nq + 1
+        step = 100000000//self.q_input.nq + 1
         #step = 1000000000
         for start in range(0, call_details.num_eval, step):
@@ -479 +479 @@
                 current_time = time.clock()
                 if current_time - last_nap > 0.5:
-                    time.sleep(0.05)
+                    time.sleep(0.001)
                     last_nap = current_time
         sync()
@@ -500 +500 @@
         Release resources associated with the kernel.
         """
-        if self.result_b is not None:
-            self.result_b.free()
-            self.result_b = None
+        for p in self._need_release:
+            p.free()
+        self._need_release = []
 
     def __del__(self):