Add ov falback to CPU machinisim & verified with OV without ROPE support#42

Closed

zhaixuejun1993 wants to merge 264 commits into

ravi9:dev_backend_openvinofrom

zhaixuejun1993:xuejun/fallback-ov-to-cpu

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 15, 2026

Update build.md and add operation mapping(GGML to OpenVINO)

zhanmyz
authored and
cavusmustafa
committed
add the rms_norm operator implemented using OpenVINO to the GGML backend of llama.cpp

zhanmyz
authored and
cavusmustafa
committed
Fix issue for output memory copy of infer request

yumengbo
authored and
cavusmustafa
committed
Change to implementation following pytorch frontend

yumengbo
authored and
cavusmustafa
committed
Add support for UNARY SILU op . Fix pytorch impl bugs.

yumengbo
authored and
cavusmustafa
committed
Support Softmax op

yumengbo
authored and
cavusmustafa
committed
Support Softmax op

yumengbo
authored and
cavusmustafa
committed
Support ROPE op.

yumengbo
authored and
cavusmustafa
committed
Add support for RMS_NORM OP

zhanmyz
authored and
cavusmustafa
committed
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML backend

zhanmyz
authored and
cavusmustafa
committed
Move CPY from GGML OV Backend to OV Frontend

zhanmyz
authored and
cavusmustafa
committed
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops

zhanmyz
authored and
cavusmustafa
committed
add implementation of CPY when the output tensor is non-contiguous

zhanmyz
authored and
cavusmustafa
committed
add tmp source code files

zhanmyz
authored and
cavusmustafa
committed
Execute singel CONT operator is OK

zhanmyz
authored and
cavusmustafa
committed
Execute CONT & VIEW operators in OV Frontend is OK

zhanmyz
authored and
cavusmustafa
committed
OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT graph conversion of consecutive OPs

zhanmyz
authored and
cavusmustafa
committed
OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT/ROPE/SCALE/SOFTMAX/ADD adjacent op graph conversion

zhanmyz
authored and
cavusmustafa
committed
Change the input parameter shape of CONT operator

zhanmyz
authored and
cavusmustafa
committed
Change the input and ouput node shape of MUL_MAT operator

zhanmyz
authored and
cavusmustafa
committed
Change the input and ouput node shape of MUL_MAT operator

zhanmyz
authored and
cavusmustafa
committed
change CONT and MULMAT input node shape

zhanmyz
authored and
cavusmustafa
committed
All adjacent ops can conversion but calculation result is wrong and need debugging

zhanmyz
authored and
cavusmustafa
committed
1. All operators implemented using OpenVINO can be successfully executed individually.

zhanmyz
authored and
cavusmustafa
committed
1. Update the implementation of CPY node when it's non-contiguous

zhanmyz
authored and
cavusmustafa
committed
Minor Update

zhanmyz
authored and
cavusmustafa
committed
Try to add VIEW node to OV Frontend and have some issues that need to be dealt with

zhanmyz
authored and
cavusmustafa
committed
1. In the Prompt process and predict first token stage, the PERMUTE node needs to be integrated into the OV Frontend

zhanmyz
authored and
cavusmustafa
committed
add debug info

zhanmyz
authored and
cavusmustafa
committed
Process Prompt and predict first token is OK

zhanmyz
authored and
cavusmustafa
committed
1. Solve the AC issue of Permute+VIEW and MULMAL issue in the phase of “1. Process Prompt and predict the first token”.

zhanmyz
authored and
cavusmustafa
committed
1. Delete some comments

zhanmyz
authored and
cavusmustafa
committed
* Use find_package in CMake to configure OpenVINO

wine99
authored and
cavusmustafa
committed
change op mappings to list in openvino_supports_op

wine99
authored and
cavusmustafa
committed
2nd+ token correct by fix CPY in OV, remove single op backend compute code

wine99
authored and
cavusmustafa
committed
Arbitrary token len (>32) work; Fix bug in mulmat

wine99
authored and
cavusmustafa
committed
FEAT: do PERMUTE eagerly

wine99
authored and
cavusmustafa
committed
FEAT: Add interleaved mode for ROPE

wine99
authored and
cavusmustafa
committed
REFACTOR: support weigts as constant

wine99
authored and
cavusmustafa
committed
STYLE: minor refactor

wine99
authored and
cavusmustafa
committed
PERF: share const nodes for weights for diff infer

wine99
authored and
cavusmustafa
committed
BUILD: update build doc, add cmake preset, add CACHE_DIR env var

wine99
authored and
cavusmustafa
committed
FEAT: improve debug capability

wine99
authored and
cavusmustafa
committed
PERF: compile once (dynamic graph + cache)

wine99
authored and
cavusmustafa
committed
Rebase - Bring up to date and fix build process

virajwad
authored and
cavusmustafa
committed
fix build error

wine99
authored and
cavusmustafa
committed
FIX: backend buffer type issue

wine99
authored and
cavusmustafa
committed
STYLE: clang-format

wine99
authored and
cavusmustafa
committed
FEAT: Add all conversion code from ov side

wine99
authored and
cavusmustafa
committed
PERF: favor low precision matmul

wine99
authored and
cavusmustafa
committed
STYLE and minor REFACTOR

wine99
authored and
cavusmustafa
committed
FIX: Re-add tensor names in cgraph, Add another case for RESHAPE

wine99
authored and
cavusmustafa
committed
FIX: input shape of KQ_mask

wine99
authored and
cavusmustafa
committed
PERF: add weight constant in parallel

wine99
authored and
cavusmustafa
committed
FIX: set_max_token_len

wine99
authored and
cavusmustafa
committed
PERF: use Slice+Concat in writing cache_v

wine99
authored and
cavusmustafa
committed
Update build doc

wine99
authored and
cavusmustafa
committed
Add cgraph tensor output name to OV op name

wine99
authored and
cavusmustafa
committed
Update openvino build instructions

ravi9
authored and
cavusmustafa
committed
Add initial NPU support

wine99
authored and
cavusmustafa
committed
draft NPU support version 2: prefill + kvcache

wine99
authored and
cavusmustafa
committed
NPU support version 2: prefill + kvcache

wine99
authored and
cavusmustafa
committed
Change due to ggml cgraph changes, not correct yet

wine99
authored and
cavusmustafa
committed
Change due to ggml cgraph changes, llama-3.2 CPU work

wine99
authored and
cavusmustafa
committed
Add AMD64 to CMakeLists

wine99
authored and
cavusmustafa
committed
Change due to ggml cgraph changes, all device work

wine99
authored and
cavusmustafa
committed
Refactor: clean, fix warning

wine99
authored and
cavusmustafa
committed
Update clang-format

wine99
authored and
cavusmustafa
committed
Statful transformation for CPU GPU

wine99
authored and
cavusmustafa
committed
Add SwiGLU

wine99
authored and
cavusmustafa
committed
Fuse to SDPA

wine99
authored and
cavusmustafa
committed
Replace Concat with Broadcast in MulMat for GQA

wine99
authored and
cavusmustafa
committed
Pull out indices creation for kv cache update

wine99
authored and
cavusmustafa
committed
Refactor: remove past_token_len from extra_inputs

wine99
authored and
cavusmustafa
committed
Fix Phi3 SwiGLU and SoftMax

wine99
authored and
cavusmustafa
committed
Pull out sin cos from rope

wine99
authored and
cavusmustafa
committed
Reduce memory: free ov weights node after graph conversion

wine99
authored and
cavusmustafa
committed
Fix CPY due to cgraph change

wine99
authored and
cavusmustafa
committed
Added OpenVINO CI/CD. Updated docs

ravi9
authored and
cavusmustafa
committed
Fix llama-cli

wine99
authored and
cavusmustafa
committed
Fix Phi3 ROPE; Add test-backend-ops

wine99
authored and
cavusmustafa
committed
Fix NPU

wine99
authored and
cavusmustafa
committed
Fix llama-bench; Clang-format

wine99
authored and
cavusmustafa
committed
Fix llama-perplexity

wine99
authored and
cavusmustafa
committed
temp. changes for mark decomp
cavusmustafa
committed
matmul in fp32

wine99
authored and
cavusmustafa
committed
mulmat input conversion fix
cavusmustafa
committed
mulmat type conversion update
cavusmustafa
committed
add mark decomp pass
cavusmustafa
committed
Revert changes in fuse_to_sdpa

wine99
authored and
cavusmustafa
committed
Update build.md

ravi9
authored and
cavusmustafa
committed
Fix test-backend-ops

wine99
authored and
cavusmustafa
committed
Skip test-thread-safety; Run ctest only in ci/run.sh

wine99
authored and
cavusmustafa
committed
Use CiD for NPU

wine99
authored and
cavusmustafa
committed
Optimize tensor conversion, improve TTFT

wine99
authored and
cavusmustafa
committed
Support op SET_ROWS

wine99
authored and
cavusmustafa
committed
Fix NPU

wine99
authored and
cavusmustafa
committed
Remove CPY

wine99
authored and
cavusmustafa
committed
Fix test-backend-ops

wine99
authored and
cavusmustafa
committed
Minor updates for raising PR

wine99
authored and
cavusmustafa
committed
Perf: RMS fused to OV internal RMS op

wine99
authored and
cavusmustafa
committed
Fix after rebasing

wine99
authored and
cavusmustafa
committed
Change openvino device_type to GPU; Enable flash_attn

wine99
authored and
cavusmustafa
committed
Update supports_buft and supports_op for quantized models

wine99
authored and
cavusmustafa
committed
Add quant weight conversion functions from genai gguf reader

wine99
authored and
cavusmustafa
committed
Quant models run with accuracy issue

wine99
authored and
cavusmustafa
committed
Fix accuracy: disable cpu_repack

wine99
authored and
cavusmustafa
committed
Fix CI; Disable test-backend-ops

wine99
authored and
cavusmustafa
committed
Fix Q4_1

wine99
authored and
cavusmustafa
committed
Fix test-backend-ops: Treat quantized tensors as weights

wine99
authored and
cavusmustafa
committed
Add NPU Q4_0 support

wine99
authored and
cavusmustafa
committed
NPU perf: eliminate zp

wine99
authored and
cavusmustafa
committed
Dequantize q4_1 q4_k q6_k for NPU

wine99
authored and
cavusmustafa
committed
Add custom quant type: q8_1_c, q4_0_128

wine99
authored and
cavusmustafa
committed
Set m_is_static=false as default in decoder

wine99
authored and
cavusmustafa
committed
Simpilfy translation of get_rows

wine99
authored and
cavusmustafa
committed
Fix after rebasing

wine99
authored and
cavusmustafa
committed
Improve debug util; Eliminate nop ReshapeReshape

wine99
authored and
cavusmustafa
committed
STYLE: make get_types_to_requant a function

wine99
authored and
cavusmustafa
committed
Support BF16 model

wine99
authored and
cavusmustafa
committed
Fix NPU compile

wine99
authored and
cavusmustafa
committed
WA for npu 1st token acc issue

wine99
authored and
cavusmustafa
committed
Apply EliminateZP only for npu

wine99
authored and
cavusmustafa
committed
Add GeGLU

wine99
authored and
cavusmustafa
committed
Fix Hunyuan

wine99
authored and
cavusmustafa
committed
Support iSWA

wine99
authored and
cavusmustafa
committed
Fix NPU accuracy

wine99
authored and
cavusmustafa
committed
Fix ROPE accuracy when freq_scale != 1

wine99
authored and
cavusmustafa
committed
Minor: not add attention_size_swa for non-swa model

wine99
authored and
cavusmustafa
committed
Minor refactor

wine99
authored and
cavusmustafa
committed
Add Q5_K to support phi-3-q4_k_m

wine99
authored and
cavusmustafa
committed
Requantize Q6_K (gs16) to gs32 on GPU

wine99
authored and
cavusmustafa
committed
Fix after rebasing

wine99
authored and
cavusmustafa
committed
Always apply Eliminate_ZP to fix GPU compile issue on some platforms

wine99
authored and
cavusmustafa
committed
kvcachefusion support
cavusmustafa
committed
env variable GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION added
cavusmustafa
committed
Fix for Phi3
cavusmustafa
committed
Fix llama-cli (need to run with --no-warmup)

wine99
authored and
cavusmustafa
committed
Fix add_sliced_mask; Revert mulmat, softmax; Remove input attention_size, iSWA model not working

wine99
authored and
cavusmustafa
committed
fix after rebasing

wine99
authored and
cavusmustafa
committed
Fix llama-3-8b and phi3-mini q4_0 NPU

wine99
authored and
cavusmustafa
committed
Update to OV-2025.3 and CMakeLists.txt

ravi9
authored and
cavusmustafa
committed
Add OV CI cache

wine99
authored and
cavusmustafa
committed
Apply CISC review and update CI to OV2025.3

ravi9
authored and
cavusmustafa
committed
Update CI to run OV dep install before build

ravi9
authored and
cavusmustafa
committed
Update OV dockerfile to use OV2025.3 and update build docs

ravi9
authored and
cavusmustafa
committed
Style: use switch in supports_ops

wine99
authored and
cavusmustafa
committed
Style: middle ptr and ref align, omit optional struct keyword

wine99
authored and
cavusmustafa
committed
NPU Unify PD (#14)

wine99
authored and
cavusmustafa
committed
Clean placeholders in ggml-openvino.cpp

wine99
authored and
cavusmustafa
committed
NPU unify PD (handled internally)

wine99
authored and
cavusmustafa
committed
change graph to 4d, support multi sequences

wine99
authored and
cavusmustafa
committed
Fix llama-bench

wine99
authored and
cavusmustafa
committed
Fix NPU

wine99
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Remove the second decoder for node. Moving the function into the model decoder

zhaixuejun1993
authored and
cavusmustafa
committed
Fix error for naive

zhaixuejun1993
authored and
cavusmustafa
committed
NPU prefill chunking

wine99
authored and
cavusmustafa
committed
NPU fix llama-bench

wine99
authored and
cavusmustafa
committed
fallback naive run with accuracy issue

wine99
authored and
cavusmustafa
committed
NPU support llma-perplexity -b 512 --no-warmup

wine99
authored and
cavusmustafa
committed
Refactor: split ov_graph_compute for dynamic and static

wine99
authored and
cavusmustafa
committed
remove unused API GgmlOvDecoder::get_output_stride(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
minor update due to ov 2025.4

wine99
authored and
cavusmustafa
committed
remove unused API GgmlOvDecoder::get_output_names()

zhaixuejun1993
authored and
cavusmustafa
committed
remove unused API get_output_shape(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Modified API GgmlOvDecoder::get_output_type(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_output_op_params(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API get_output_ggml_tensor(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API m_outputs

zhaixuejun1993
authored and
cavusmustafa
committed
Removed m_output_names

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_names()

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_stride(const std::string& name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API get_input_type

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API get_input_type

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_shape(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_op_params(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Fix error for decoder cache

zhaixuejun1993
authored and
cavusmustafa
committed
Reuse cached decoder

wine99
authored and
cavusmustafa
committed
GPU remove Q6_K requantization

wine99
authored and
cavusmustafa
committed
NPU fix wrong model output shape

wine99
authored and
cavusmustafa
committed
NPU fix q4 perf regression

wine99
authored and
cavusmustafa
committed
Remove unused variable nodes

zhaixuejun1993
authored and
cavusmustafa
committed
Fix decoder can_reuse for llama-bench

wine99
authored and
cavusmustafa
committed
Update build.md for Windows

I-N-T-E-L
authored and
cavusmustafa
committed
backend buffer: allocate on host

wine99
authored and
cavusmustafa
committed
Use shared_buffer for GPU NPU; Refactor

wine99
authored and
cavusmustafa
committed
Add ov_backend_host_buffer; Use cached remote context

wine99
authored and
cavusmustafa
committed
Put kvcache on GPU

wine99
authored and
cavusmustafa
committed
Use ggml_aligned_malloc

wine99
authored and
cavusmustafa
committed
only use remote tensor for kvcache

wine99
authored and
cavusmustafa
committed
only use remote tensor for kvcache for GPU

wine99
authored and
cavusmustafa
committed
FIX: use remote tensor from singleton

wine99
authored and
cavusmustafa
committed
Update build.md to include OpenCL

wine99
authored and
cavusmustafa
committed
NPU always requant to q4_0_128

wine99
authored and
cavusmustafa
committed
Optimize symmetric quant weight extraction: use single zp

wine99
authored and
cavusmustafa
committed
Use Q8_0_C in token embd, lm_head, and for 5 and 6 bits quant

wine99
authored and
cavusmustafa
committed
Update build.md

wine99
authored and
cavusmustafa
committed
Support -ctk f32

wine99
authored and
cavusmustafa
committed
Initial stateful graph support
cavusmustafa
committed
Update ggml/src/ggml-openvino/ggml-decoder.cpp

cavusmustafa
and
ynimmaga
committed
code cleanup
cavusmustafa
committed
npu perf fix
cavusmustafa
committed
requant to f16 for Q6 embed on NPU
cavusmustafa
committed
Update ggml/src/ggml-openvino/ggml-decoder.cpp
cavusmustafa
committed
Update ggml/src/ggml-openvino/ggml-openvino-extra.cpp
cavusmustafa
committed
Create OPENVINO.md in llama.cpp backend docs

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update build.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
kq_mask naming fix
cavusmustafa
committed

Commits on Jan 16, 2026

Syntax correction for workflows build file
cavusmustafa
committed

Commits on Jan 21, 2026

Commits on Jan 22, 2026

Commits on Jan 24, 2026

Don't put kvcache on GPU in stateful mode
wine99
committed

Commits on Jan 27, 2026

Remove hardcode names
wine99
committed
Fix stateful shapes
wine99
committed
Simplification for stateful and update output shape processing

cavusmustafa
authored and
wine99
committed

Commits on Feb 5, 2026

Commits on Feb 6, 2026

Refactor weight tensor processing
wine99
committed

Commits on Feb 9, 2026

Add ov falback to CPU machinisim & verified with OV without ROPE support

zhaixuejun1993
authored and
cavusmustafa
committed