[API] Filter refactor

[SchrodingersGat]

This PR is a major refactor of the enable_filter functionality which allows optional fields to be added (or removed) dynamically to API serializers.

Problem Description

The old code was written in such a way that the optional serializer class was instantiated on definition.

Consider the following code

    category_detail = enable_filter(
        CategorySerializer(
            source='category', many=False, read_only=True, allow_null=True
        ),
        prefetch_fields=['category'],
    )

The CategorySerializer class is instantiated (which is quite expensive) - and then this is also chained down for any multi-level child serializers.

In many cases, the fields are later removed, so instantiating them early is a waste of resources.

Also, due to the (wasteful?) way that DRF deep-copies all the fields (multiple times throughout the lifespan of the serializer) this can be very prohibitive.

Some particularly bad API endpoints which had deep nested serializers, saw hundreds of thousands of serializer objects created, and then later deleted.

Solution

The PR introduces an OptionalField approach, which lazily evaluates the optional fields only after we have decided that they should definitely be included in the serializer:

    category_detail = OptionalField(
        serializer_class=CategorySerializer,
        serializer_kwargs={
            'source': 'category',
            'many': False,
            'read_only': True,
            'allow_null': True,
        },
        prefetch_fields=['category'],
    )

Justification

The new API endpoints are significantly faster, we have been introducing a huge amount of wasted overhead (for years now in the codebase) due to the unnecessary serializer evaluation

Benchmarks

Benchmarking shows that serializers which are deeply nested have the most benefit from this PR. Both GET and OPTIONS requests are improved substantially by deferring serializer instantiation.

Methodology

• Perform external requests via python API bindings
• Average across 100 requests per test
• All times specified in ms

GET

• limit=50
• offset=0

URL	Master	PR
/api/part/	151	89
/api/part/category/	43	43
/api/stock/	173	74
/api/stock/location/	64	42
/api/company/	47	38
/api/build/	86	47
/api/build/line/	646	351
/api/build/item/	276	41
/api/order/so/	75	50
/api/order/so/shipment/	90	79
/api/order/po/	89	50
/api/order/po-line/	187	62
/api/user/roles/	32	37
/api/parameter/	42	36
/api/parameter/template/	34	33

OPTIONS

URL	Master	PR
/api/part/	85	71
/api/part/category/	43	33
/api/stock/location/	42	37
/api/company/	43	36
/api/build/	72	61
/api/build/line/	266	44
/api/build/item/	210	31
/api/order/so/	67	60
/api/order/so/shipment/	70	50
/api/order/po/	66	57
/api/order/po-line/	185	60
/api/user/roles/	28	30
/api/parameter/	36	35
/api/parameter/template/	30	33

[netlify]

✅ Deploy Preview for inventree-web-pui-preview canceled.

Name	Link
🔨 Latest commit	95e0aa8
🔍 Latest deploy log	https://app.netlify.com/projects/inventree-web-pui-preview/deploys/695858dfbc24630008f6f9c3

[Copilot]

Pull request overview

This PR refactors the enable_filter functionality to use a new OptionalField dataclass approach for lazy evaluation of optional serializer fields, addressing performance issues caused by premature serializer instantiation.

Key changes:

• Replaced enable_filter() function with OptionalField dataclass for declarative field definitions
• Refactored FilterableSerializerMixin to lazily instantiate optional fields only when needed
• Updated all serializers across the codebase to use the new OptionalField pattern

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
InvenTree/serializers.py	Introduced OptionalField dataclass and refactored FilterableSerializerMixin to support lazy field initialization via build_unknown_field()
InvenTree/test_serializers.py	Updated tests to use new OptionalField API; removed obsolete enable_filter validation test; corrected spelling from "failiure" to "failure"
users/serializers.py	Migrated GroupSerializer fields (permissions, roles, users) from enable_filter to OptionalField
stock/serializers.py	Converted multiple optional detail fields including user_detail, template_detail, location_path, part_detail, supplier_part_detail, tests, item_detail to use OptionalField
part/serializers.py	Refactored optional fields across CategorySerializer, PartBriefSerializer, PartSerializer, BomItemSerializer, and related serializers to use OptionalField pattern
order/serializers.py	Updated order-related serializers including purchase order, sales order, and return order serializers to use OptionalField for detail fields
company/serializers.py	Migrated company, manufacturer part, and supplier part serializers' optional fields to OptionalField
common/serializers.py	Converted parameter serializer detail fields to use OptionalField
common/filters.py	Updated filter helper functions to return OptionalField instances instead of enable_filter wrapped fields
build/serializers.py	Refactored build serializer optional fields including part, user, and item detail fields to use OptionalField

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 97.83394% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.19%. Comparing base (97dd664) to head (95e0aa8).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #11073   +/-   ##
=======================================
  Coverage   88.18%   88.19%           
=======================================
  Files        1290     1290           
  Lines       58142    58225   +83     
  Branches     1969     1969           
=======================================
+ Hits        51272    51349   +77     
- Misses       6379     6385    +6     
  Partials      491      491           

Flag	Coverage Δ
backend	89.48% <97.83%> (+<0.01%)	⬆️
migrations	42.41% <70.03%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Backend Apps	92.03% <100.00%> (+0.01%)	⬆️
Backend General	93.48% <100.00%> (ø)
Frontend	70.85% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[matmair]

I might be overlooking something, but are we not completely loosing static introspection with this change? I am not sure if there is much actual runtime performance to be gained by this, have you benchmarked it?

[SchrodingersGat]

I am not sure if there is much actual runtime performance to be gained by this, have you benchmarked it?

Benchmarks added to the top comment.

I might be overlooking something, but are we not completely loosing static introspection with this change?

We should not lose any introspection as the fields are still created - they are just defered until we determine they are actually needed. This is most important in the case of deeply nested serializer fields which will never be exposed to the final serializer tree.

The OPTIONS endpoints still show all the optional fields, and I believe the API docs should be the same. LMK if you spot any items which are no longer included.