Skip to content

Enable OpenVINO backend fallback to CPU backend#184

Open
zhaixuejun1993 wants to merge 2 commits into
ravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/ov-fallabck-cpu
Open

Enable OpenVINO backend fallback to CPU backend#184
zhaixuejun1993 wants to merge 2 commits into
ravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/ov-fallabck-cpu

Conversation

@zhaixuejun1993
Copy link
Copy Markdown
Collaborator

This pull request introduces enhancements to the tensor tracking and backend scheduling mechanisms in the GGML library. The main changes add support for tracking original source tensors during in-place operations, improve backend assignment logic, and enhance debugging and dynamic dimension computation. These improvements increase the robustness and debuggability of tensor operations, especially in complex backend and view scenarios.

Tensor tracking and metadata:

  • Added a new org_src pointer to the ggml_tensor struct to keep track of original source tensors in in-place operations, and initialized it to NULL in tensor creation (ggml/include/ggml.h, ggml/src/ggml.c). [1] [2]

Backend scheduling and assignment:

  • Improved backend assignment logic in ggml_backend_sched_split_graph to check if a view source's backend supports the operation before assigning its backend ID to the current tensor, and to propagate backend IDs to view ops when appropriate (ggml/src/ggml-backend.cpp). [1] [2]
  • When copying tensors for backend splitting, the org_src field is now set to the source tensor for better tracking (ggml/src/ggml-backend.cpp).

Debugging and diagnostics:

  • Appended node IDs to tensor names during backend scheduling for easier debugging (ggml/src/ggml-backend.cpp).

Dynamic dimension handling:

  • Updated the OpenVINO decoder to utilize the org_src field for root source tracking and added checks to ensure dynamic dimension logic only applies to tensors without an original source (ggml/src/ggml-openvino/ggml-decoder.cpp, ggml/src/ggml-openvino/ggml-decoder.h). [1] [2] [3]## Overview

Additional information

Requirements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant