Missing quantization type

[damian1996]

class BitPrecision(Enum):
    none = 'none'
    int8 = 'int8'
    int4 = 'int4'
    float16 = 'float16'
    float32 = 'float32'


class Method(Enum):
    None_ = 'None'
    dynamic = 'dynamic'
    static = 'static'


class Quantization(BaseModel):
    bit_precision: BitPrecision = Field(..., description='Quantization bit precision')
    method: Method = Field(..., description='Quantization method')

We should add type field to Quantization class because the model results depends on the quant type (for example qptq, awq and so on).

[borgr]

Agreed. How diverse are those? those are not rapidly changing and an enum should be fine, right?

[damian1996]

I think so. There are a few basic quantization techniques like AWQ, PTQ, GPTQ, QAT, BitsAndBytes. Enum should be sufficient.

[damian1996]

What do you think about adding quantization suffix here? I'm not sure if it is related only to GGUF quantization (for other quantization types I have not seen these suffices in HF. Maybe adding it as optional field would be a good option.

[damian1996]

There is also a issue with bit precision allowed for a quantization. In above screen, we can see 5-bit or 6-bit for example and our current schema did not account for it.

[borgr]

I don't understand the suggestion, When you say "suffix", you mean a class variable with the number of bits?

[damian1996]

By suffix I meant e.g. 'Q3_K_S', sorry for confusing naming

[borgr]

We surely want to separate different quantizations, yes. They do it in string form, we don't (we can always add a syntactic sugar of string to class and class to string, if this get's annoying)