Improve `topic_key` generation: sorted segments + hierarchical sub-paths

[carlosjortiz]

📋 Pre-flight Checks

• I have searched existing issues and this is not a duplicate
• I understand this issue needs status:approved before a PR can be opened

🔍 Problem Description

The current topic_key generation in SuggestTopicKey() has two limitations:

1. Word order sensitivity causes silent duplicates

The segment portion of the key preserves the original word order from the title. This means observations about the same topic can produce different keys depending on phrasing:

• "Auth model refactor" → architecture/auth-model-refactor
• "Refactor auth model" → architecture/refactor-auth-model

Since topic_key drives upsert logic, these become two separate observations instead of updating the same one. Over time this silently fragments memory, especially with AI agents as primary writers.

2. Flat structure limits organization

The current format is family/segment (e.g., architecture/auth-model). There's no way to group related observations under a domain:

• architecture/auth-model and architecture/auth-middleware have no explicit relationship
• Querying "everything about auth architecture" requires FTS, not a simple prefix match

💡 Proposed Solution

A. Sort words alphabetically in segment generation

In normalizeTopicSegment(), sort the words before joining:

words := strings.Fields(strings.ToLower(segment))
sort.Strings(words)
segment = strings.Join(words, "-")

Before: "Auth model refactor" → auth-model-refactor
After: "Auth model refactor" → auth-model-refactor (same)
After: "Refactor auth model" → auth-model-refactor (same! was different before)

This makes upsert matching robust regardless of how the agent phrases the title.

B. Support hierarchical sub-paths (family/domain/segment)

Allow an optional middle level in the key hierarchy:

architecture/auth/model
architecture/auth/middleware
bug/payments/nil-panic
decision/api/versioning-strategy

This enables prefix queries like:

SELECT * FROM observations WHERE topic_key LIKE 'architecture/auth/%'

No schema change required — topic_key is already a TEXT column. This is purely a convention + generation logic change.

📦 Affected Area

Store (database, queries)

🔄 Alternatives Considered

Considered adding a tags column with a junction table for flexible categorization. Discarded because the primary writers are AI agents, which are inconsistent at tagging (e.g., auth vs authentication vs login). Without a human curation loop, tags would add noise rather than improve search. FTS5 + topic_key already covers the search use case without the extra schema complexity.

📎 Additional Context

Recommendation

I recommend implementing both solutions together — they complement each other well:

• A (sorted segments) eliminates silent duplicates at the source
• B (sub-paths) adds the organizational layer that makes prefix queries useful

Implementing them separately would work, but together they deliver a cohesive improvement to topic_key reliability and discoverability.

Volunteer

I'm happy to take this on and submit a PR for both changes. The scope is well-defined and contained.

Impact

• Reduces duplicate observations caused by inconsistent agent phrasing
• Improves memory organization with hierarchical grouping
• Zero schema changes — works within existing TEXT column
• Low implementation effort — core change is ~10 lines in normalizeTopicSegment() + convention updates

[avirgili-eclub]

Interesting.. It may be worth considering whether sorting segment words alphabetically changes the role of topic_key from a human-shaped canonical slug into a machine-canonical token set.

It seems topic_key in this project is not purely internal or limited to storage details. It is part of the observation model, accepted through the API, included in serialized/exported data, and used by the store’s upsert path to decide whether a new save should update an existing observation in the same project + scope + topic_key group.

Because of that, I think that changing the generation logic here would not only improve deduplication behavior, but also effectively redefine the canonical ID convention used across storage, API input/output, and exported data. That makes the tradeoff a bit broader than just “avoiding silent duplicates”.

It may also be worth considering whether hierarchical subpaths are already at least partially possible today if callers provide them explicitly. Since topic_key is stored as text and used as-is for matching, the missing piece may not be storage support itself, but rather consistent generation and documentation of that convention. Something like architecture/auth/model may already be viable when supplied by the caller, even if the current suggestion/generation path does not produce it consistently yet.

There also seems to be a documentation consistency issue around the expected format. Some examples use slash-based keys such as architecture/auth-model, while docs/ARCHITECTURE.md shows architecture-auth-architecture. It may help to align the intended convention first, then decide whether alphabetical sorting fits that convention or whether it would be better to keep canonical matching concerns separate from human-readable naming.

[[Edited with AI for clarity since English is not my first language.]]

[carlosjortiz]

Thanks for the thorough analysis — these are all valid points.

I'm aware that this change has broader implications beyond just deduplication. topic_key touches the API surface, exported data, and the upsert path, so redefining how it's generated is effectively redefining the canonical ID convention system-wide.

That's intentional. The way I see it, this would be the first step in a two-part effort:

1. This PR — align the generation logic (sorted segments + sub-path support) so that the engine produces consistent, hierarchical keys.
2. Follow-up in gentle-ai — update the conventions, plugin instructions, and documentation to match the new canonical format, ensuring callers and agents produce keys that align with the new generation logic.

Regarding your point about sub-paths already working today: you're right, the storage layer already supports them since it's just TEXT matching. The gap is exactly what you identified — consistent generation and clear documentation of the convention. That's what this change aims to close.

On the documentation inconsistency (architecture/auth-model vs architecture-auth-architecture): agreed, that needs to be resolved as part of this work. The slash-based hierarchical format should be the canonical convention going forward.