1317 applicationservice fails to access applications inside private folders

[nicolasbisurgi]

Summary

Problem

The ApplicationService failed with 404 errors when accessing applications inside private folders because the TM1 REST API requires PrivateContents('FolderName') for private folder segments, but TM1py was always using Contents('FolderName') for intermediate path segments.

Solution: Implemented automatic path resolution with an optimistic probing strategy:

1. Optimistic probing - Try all-public path first (most common case), then all-private path
2. Fallback discovery - If both fail, iteratively probe each segment to find the exact public-to-private transition point
3. Efficient probing - Uses $top=0 query parameter to probe paths without fetching actual data
4. Optional caching - New use_cache parameter (default False) to cache discovered private boundaries for repeated access
5. User warnings - Emits warnings when auto-resolution is used so users are aware of the path correction

Changes:

• Added _build_path_url() helper to construct URL paths with mixed Contents/PrivateContents segments
• Added _find_private_boundary() to iteratively discover where a path becomes private
• Added _resolve_path() that implements the optimistic probing strategy
• Updated all public methods (get_names, get, exists, delete, rename, create, update, etc.) to use path resolution
• Added use_cache parameter to all affected methods
• Added comprehensive unit tests for the new functionality

[MariusWirtz]

The optimistic approach is very sensible. I like it. The costs for the iterations are acceptable IMO.

However, I noticed that the new code breaks some tests that run fine in the master branch

I think the rename function requires some additional handling, as the URLs with tm1.Move extension don't support GET.

[nicolasbisurgi]

Summary

• Fixes ApplicationService fails to access applications inside private folders #1317 - ApplicationService now correctly accesses applications inside private folders
• Added new discover() method to explore the Applications folder hierarchy
• Refactored path resolution to handle mixed public/private folder structures

New Features

discover() Method

Explore the Applications folder and discover all items including private assets:

# Discover all items at root level
items = tm1.applications.discover(path="", include_private=True)

# Discover recursively with nested structure
items = tm1.applications.discover(
    path="Planning Sample", 
    include_private=True, 
    recursive=True,
    flat=False  # Returns nested structure with 'children' key
)

# Each item contains: @odata.type, type, id, name, path, is_private

Copy Private Document to Public Location

from TM1py import TM1Service
from TM1py.Objects.Application import DocumentApplication

with TM1Service(**params) as tm1:
    # Get the private document
    private_doc = tm1.applications.get_document(
        path="Planning Sample/Administrator/PrivateFolder",
        name="My Private Document",
        private=True
    )
    
    # Create as public document
    public_doc = DocumentApplication(
        path="Planning Sample/Administrator",
        name="My Public Document",
        content=private_doc.content
    )
    
    tm1.applications.update_or_create(application=public_doc, private=False)

Access Documents in Nested Private Folders

# Now works correctly - previously failed with 404
doc = tm1.applications.get_document(
    path="Planning Sample/Administrator/PrivateFolder/SubFolder",
    name="Nested Document",
    private=True
)

Test Plan

• All existing ApplicationService tests pass
• Added tests for _build_path_url() with various boundary scenarios
• Added tests for discover() (public, private, recursive, flat modes)
• Added tests for document operations in private folders
• Added tests for nested private folder structures
• Added tests for copying documents between private/public locations

[github-actions]

Tests completed for environment: tm1-11-cloud. Check artifacts for details.

[MariusWirtz]

Looks really good. Thank you, @nicolasbisurgi.

I hesitate a little bit with the private boundary cache mechanic.
TM1py so far is stateless. Doesn't this cache represent a minor change in approach?
What is the price for not caching? Is it a measurable loss in performance? Or too many micro requests that trigger PAoC remote disconnects?

No strong opinion, I just wanted to share my thoughts on this one.
@onefloid, what do you think?

[onefloid]

1. Before we go further: is the trial-and-error approach via API calls the only way to determine whether a path is public or private, or is there any existing (possibly undocumented) API capability that we could leverage instead? @Hubert-Heijkers

2. In my opinion, introducing a small cache here is a common and pragmatic optimization and does not fundamentally break statelessness.

3. I would suggest changing the search strategy in order to reduce the number of API calls.

   Currently, boundary finding is implemented as a linear scan over the path segments. Since the privacy property along the path is monotonic (public → private → private …), we can instead apply a divide-and-conquer approach using a binary boundary search.

   This allows us to find the first private path segment in O(log n) API calls instead of O(n), while preserving the same functional behavior.

[nicolasbisurgi]

Thanks for the feedback!

Approach:

I do agree having @Hubert-Heijkers insight on this would be great. I was a bit surprised to see that there was no way to get this information without a trial an error approach.

Caching

Regarding caching, I think that if we leave it False as default we should still be considering TM1Py stateless. The reason I'm working on this topic is to create a script to transform all private applications into public ones for <500 user model, so having the ability to cache some of the valid mixed paths was a nice to have

Strategy

I agree, binary approach is more efficient but we would start seeing better performance starting on 4 level deep onwards, right? This might be more useful on v12 than v11 considering how Cloudfare handles repeated requests.

Next steps Proposal

I think we can proceed as is and wait for a response from Hubert; depending on where that answer leaves us we can optimize this approach by replacing linear by binary.

Thoughts?

[MariusWirtz]

Yeah. I think the search optimization can be done in a future PR, given the expected performance impact.

Thanks everyone, for the feedback on this!

[Hubert-Heijkers]

@onefloid, @nicolasbisurgi, correct me if I'm wrong but my (quick) understanding here is that in TM1py any path, which end in a public or private target document, gets dumbed down to a simple [file] path, effectively ignoring that one little aspect that the same name can actually resolve in both a public as well as a private path or document. Truth be told this has never been one of my favorite things in TM1 either, not in applications, and especially not in public/private subsets/views and the implicit behavior that stems from it.

The iterative resolve of such a path not knowing if a segment represents a public or private entry only works if only either a public or private entry with such a name exists. But if that were always the case we wouldn't have had to have separate public (Contents) and private (PrivateContents) collections to begin with. I.o.w. the conversion to a path is not lossless.

As for resolving one could in theory resolve the entry, or more entries if multiple existed with the same 'path', using one REST request IF TM1 supported type cast segments on these entries (which I just got reminded it does not - yet). A request for a path like TM1py/PR-test/aaa would then look something like:

GET ~/Contents('Applications')/Contents
    ?$filter=Name eq 'TM1py' and tm1.Folder/Contents/any(c:c/tm1.Folder and c/Name eq 'PR-test' and c/tm1Folder/Contents/any(d:d/Name eq 'aaa')) or and tm1.Folder/PrivateContents/any(c:c/tm1.Folder and c/Name eq 'PR-test' and c/tm1Folder/Contents/any(d:d/Name eq 'aaa'))
    &$expand=tm1.Folder/Contents(
        $filter=Name eq 'PR-test' and (tm1.Folder/Contents/any(c:not c/tm1.Folder and c/Name eq 'aaa') or tm1.Folder/PrivateContents/any(c:not c/tm1.Folder and c/Name eq 'aaa'));
        $expand=tm1.Folder/Contents($filter=Name eq 'aaa'),tm1.Folder/PrivateContents($filter=Name eq 'aaa'));
      $expand=tm1.Folder/PrivateContents(
        $filter=Name eq 'PR-test' and tm1.Folder/PrivateContents/any(c:not c/tm1.Folder and c/Name eq 'aaa');
        $expand=tm1.Folder/PrivateContents($filter=Name eq 'aaa'))

This can probably be a bit more optimize where we know we have to be dealing with a folder anyway, or have to be, but it's all mute until the type segment on the collection and entry types is supported unfortunately and even then this would give you one or more entries and you'd likely still not know which one right?
Long story short I think there is something more fundamental that needs to fixed, the path used to identify an entry in the Application folders.