Deployment configurationslink

IREE provides a flexible set of tools for various deployment scenarios. Fully featured environments can use IREE to load programs on demand and to take advantage of multi-threaded hardware, while embedded systems can bypass IREE's runtime entirely or interface with custom accelerators.

Stable configurationslink

CPU for general purpose CPU deployment
CPU - Bare-Metal with minimal platform dependencies
GPU - Vulkan for cross-platform usage and interop with graphics applications
GPU - ROCm for AMD-specific solutions
GPU - CUDA for NVIDIA-specific solutions
GPU - Metal for running on Apple hardware

These are just the most stable configurations IREE supports. Feel free to reach out on any of IREE's communication channels if you have questions about a specific platform, hardware accelerator, or set of system features.

Compiler target backendslink

Compiler target backends are used to generate executable code for hardware APIs and device architectures. Compiler targets may implement special optimizations or generate distinct code for certain device/architecture/performance profiles.

When compiling programs, a list of target backends must be specified via

--iree-hal-target-backends= (command-line)
target_backends=[...] (Python)

Target backend	Description	Compatible HAL devices
`llvm-cpu`	Code generation for CPU-like devices supported by LLVM	`local-sync`, `local-task`
`vmvx`	Portable interpreter powered by a microkernel library	`local-sync`, `local-task`
`vulkan-spirv`	Portable GPU support via SPIR-V for Vulkan	`vulkan`
`rocm`	AMD GPU support via HSACO for HIP	`hip`
`cuda`	NVIDIA GPU support via PTX for CUDA	`cuda`
`metal-spirv`	GPU support on Apple platforms via MSL for Metal	`metal`
`webgpu-spirv`	Experimental GPU support on the Web via WGSL for WebGPU	`webgpu`

Listing available backendslink

The list of compiler target backends can be queried:

Command-linePython bindings

$ iree-compile --iree-hal-list-target-backends

Registered target backends:
    cuda
    llvm-cpu
    metal-spirv
    rocm
    vmvx
    vmvx-inline
    vulkan-spirv

import iree.compiler as ireec

ireec.query_available_targets()
# ['cuda', 'llvm-cpu', 'metal-spirv', 'rocm', 'vmvx', 'vmvx-inline', 'vulkan-spirv']

Runtime HAL drivers and deviceslink

Runtime HAL drivers can be used to enumerate and create HAL devices.

Runtime HAL devices call into hardware APIs to load and run executable code. Devices may use multithreading or other system resources, depending on their focus and the build configuration.

HAL device	Description
`local-sync`	Synchronous local CPU device with inline execution
`local-task`	Multithreaded local CPU device using a 'task' executor
`vulkan`	Portable GPU execution using the Vulkan API
`hip`	AMD GPU execution using HIP
`cuda`	NVIDIA GPU execution using CUDA
`metal`	GPU execution on Apple platforms using Metal
`webgpu`	Experimental GPU execution on the web using WebGPU

Tip - External HAL drivers

Additional HAL drivers can also be defined out of tree via the IREE_EXTERNAL_HAL_DRIVERS CMake option.

Listing available drivers and deviceslink

The list of runtime HAL drivers and devices can be queried:

Command-linePython bindings

List drivers:

$ iree-run-module --list_drivers

# ============================================================================
# Available HAL drivers
# ============================================================================
# Use --list_devices={driver name} to enumerate available devices.

        cuda: NVIDIA CUDA HAL driver (via dylib)
         hip: HIP HAL driver (via dylib)
  local-sync: Local execution using a lightweight inline synchronous queue
  local-task: Local execution using the IREE multithreading task system
      vulkan: Vulkan 1.x (dynamic)

List devices:

$ iree-run-module --list_devices

hip://GPU-00000000-1111-2222-3333-444444444444
local-sync://
local-task://
vulkan://00000000-1111-2222-3333-444444444444

Dump information about devices:

$ iree-run-module --dump_devices

# ============================================================================
# Enumerated devices for driver 'cuda'
# ============================================================================


# ============================================================================
# Enumerated devices for driver 'hip'
# ============================================================================

# ===----------------------------------------------------------------------===
# --device=hip://GPU-00000000-1111-2222-3333-444444444444
#   AMD Radeon PRO W7900 Dual Slot
# ===----------------------------------------------------------------------===

- amdhip64_dylib_path: /opt/rocm-6.1.3/lib/libamdhip64.so
- gpu-compute-capability: 11.0
- gpu-arch-name: gfx1100

- launch-max-block-dims: (1024, 1024, 1024)

- block-max-thread-count: 1024
- block-max-32-bit-register-count: 65536
- block-max-shared-memory: 64 KB

- memory-is-integrated-memory: 0
- memory-supports-managed-memory: 1
- memory-total-const-memory-size: 2047 MB
- memory-total-global-memory-size: 46064 MB
- memory-l2-cache-size: 6291456 bytes

- gpu-compute-unit-count: 48
- gpu-compute-max-clock-rate: 1760 mHz
- gpu-memory-max-clock-rate: 1124 mHz
- gpu-warp-size: 32
# ============================================================================
# Enumerated devices for driver 'local-sync'
# ============================================================================

# ===----------------------------------------------------------------------===
# --device=local-sync://
#   default
# ===----------------------------------------------------------------------===

# ============================================================================
# Enumerated devices for driver 'local-task'
# ============================================================================

# ===----------------------------------------------------------------------===
# --device=local-task://
#   default
# ===----------------------------------------------------------------------===

# ============================================================================
# Enumerated devices for driver 'vulkan'
# ============================================================================

# ===----------------------------------------------------------------------===
# --device=vulkan://00000000-1111-2222-3333-444444444444
#   AMD Radeon PRO W7900 Dual Slot  (RADV GFX1100)
# ===----------------------------------------------------------------------===

List drivers:

import iree.runtime as ireert

ireert.system_setup.query_available_drivers()
# ['cuda', 'hip', 'local-sync', 'local-task', 'vulkan']

List devices:

import iree.runtime as ireert

for driver_name in ireert.system_setup.query_available_drivers():
    print(driver_name)
    try:
        driver = ireert.get_driver(driver_name)
        device_infos = driver.query_available_devices()
        for device_info in device_infos:
            print(f"  {device_info}")
    except:
        print(f"  (failed to initialize)")

# cuda
#   (failed to initialize)
# hip
#   {'device_id': 1, 'path': 'GPU-00000000-1111-2222-3333-444444444444', 'name': 'AMD Radeon ...'}
# local-sync
#   {'device_id': 0, 'path': '', 'name': 'default'}
# local-task
#   {'device_id': 0, 'path': '', 'name': 'default'}
# vulkan
#   {'device_id': 1234, 'path': '00000000-1111-2222-3333-444444444444', 'name': 'AMD Radeon ...'}