Plan of action for song search release #67
Description
Overview
This issue outlines a plan of action to release the song search feature (#44), including a detailed breakdown of the remaining steps to finish a minimum viable implementation.
Previous pull requests (especially #53 and #59) have done a lot of the groundwork - both through some very necessary refactoring and through making the song API endpoints more flexible to support the addition of search parameters. These have been merged to the develop branch to prevent their changes from becoming stale and avoid long-lived branches.
But right now, we're a bit stuck in scope creep, doing a lot of work in different parts that aren't necessarily contributing to a cohesive release. My goal with this write-up is to outline the scope we'll be working with for the initial search feature's release, and then convert that rationale into a set of actionable tasks that we can follow toward its completion.
Everything below is written within the context of this commit.
Rationale
Below, we bring some design considerations that justify the necessary changes. It also outlines some things we will not be doing as part of this initial effort, which will also help us delimit the scope of this stage.
Filters
We've been working on designing a highly advanced set of filters in this Excalidraw board. This was one of the initial versions which we submitted in our Discord:
Below is a refined design based on community feedback and inspired by the Modrinth app:
These filters are currently only partially implemented in the API; the frontend is nowhere near functional at the moment.
With a significant portion of the work toward a functional search already complete, the website would benefit from us releasing it sooner rather than later. A basic search bar, even without advanced filtering, is already miles better than no search at all. Many users have been resorting to Note Block World Finder (a community-made website that offers basic search capabilities) or even leveraging Discord's search to find it in our server's feed channel.
We have the power to improve this situation right now. As such, I believe our best course of action is to postpone the release of filters to a secondary stage and ship a functional search feature with basic sorting as soon as possible.
Search engine
As it turns out, search options for Mongo databases are quite limited. When hosted in Atlas, their managed database service, we can leverage Atlas Search to do the heavy-lifting for us. But since this is not our case, we have to resort to good old text search within the DB itself.
Unfortunately, this approach is quite limited. Without fuzzy matching, we can only search based on exact matches. For instance, the search string "Volume Alpha" doesn't match songs containing "Alpha" unless we split the words into individual queries (which we have resorted to in our current implementation). It also doesn't account for relevance or similarity during matching, which prevents a song with a typo from appearing in search results.
In order to offer more powerful search capabilities in the future, we plan to transition to a fully fledged search engine, such as Elasticsearch (commercial) or, likely, Typesense (OSS). Its implementation is not trivial, requiring another server to be up and running and a complex strategy to keep the data synchronized (not implementing it correctly could cause an array of problems, for instance, deleted songs reappearing in search results!).
This will significantly increase the complexity of both hosting and maintaining the website and, as such, is being postponed to an undefined future date.
Server-side rendering
Right now, search is implemented entirely in the frontend. We navigate to the /search page and a loading skeleton appears while the results are fetched. If we compare this approach with YouTube, for instance, we see that we only navigate to the page after the results are ready. That's because the page is entirely rendered on the server, with the user waiting on the same page until the result page is ready to be viewed.
In our case, it would provide a better user experience to keep what the user is already viewing on the screen instead of leading them to a different page, only to make them wait for the search results to appear. As such, we'd ideally resort to server-side rendering (as we're doing with a few parts of the website already) before releasing the feature, as this is a large structural change in the implementation that could be more troublesome to ship later.
Pagination
Right now, we're using a 'Load more' button (similar to the homepage's recent songs) to load more results. This approach has been chosen to give users time to process what they're seeing at a given moment. It's easy to be overwhelmed by an endless stream of options when they load without the user's control -- a clickable button gives users time to absorb what's on screen before being presented with even further options.
There are two alternatives to this approach that I'd like to explore: infinite scrolling and true pages.
When browsing a list of recent songs, users often don't know what they want - they don't arrive in that section with a specific goal in mind. The idea, then, is for them to be able to skim through the options to find something that piques their interest. In a search page, however, the user already comes with a clear idea of what they're looking for. They know exactly what they have in mind and are just looking for the closest match. For this reason, the way we present this information to make it most useful can be significantly different from other types of views.
YouTube's approach to its search pages is infinite scrolling. We're also looking at the Modrinth app as a reference implementation, and they chose to go with pages instead:
Infinite scrolling has the drawback of making deterministic URLs impractical. If you'd like to browse to page number 100, you have to scroll to the bottom of the page 100 times. Copying a URL that points to this part of the page would cause 100 pages worth of content to be fetched immediately upon load, something that can only be prevented with proper segmentation in pages (like Modrinth does). This way, visiting page 100 skips the first 99 pages and loads content only from that point onward.
However, we also have to consider the differences in the type of content that each option hosts. A datapack or mod is a long-term project and each individual result is a much larger effort compared to a single song. As such, endless scrolling wouldn't be suitable or relevant; instead, the ability to be precise in your search is much more valuable in that case.
Page numbers are not indispensable in our case, though - we can get away without them, which lets us consider infinite scrolling as an option.
One consideration with infinite scrolling is that the footer of the page becomes inaccessible until results are exhausted. I don't believe this would be much of a problem in our current scenario, since there aren't many songs on the website to browse through - most search queries wouldn't even fill an entire page worth of results. In a scenario where this would be more significant, such as YouTube, often it's necessary to redesign the whole page so the footer elements are accessible from somewhere else.
For simplicity of implementation, consistency with the 'Recent songs' page, release speed, and to avoid tackling the unreachable-footer problem now, we can go with a 'Load more' button identical to the one in the 'Recent songs' page, for the time being.
Content display
Lastly, for the same reason we mentioned above, the way content is displayed often changes between regular browsing vs. a search page. For someone browsing the front page, it's likely not relevant to learn about all the details of each song you're viewing. In a search page, however, the user is, quite likely, narrowing their choice down to a single song or a few options. As such, it's suitable to display more detailed/dense information compared to the regular song cards.
This is the case on YouTube where, for instance, search results display a snippet of the description, the channel's profile picture, and additional badges. You're no longer limited to just skimming the content to see what catches your interest; now, you're actively interested in knowing what's in that video to decide whether or not it's the one you'd like to click next. To present this information, YouTube uses a 'list view' with vertically arranged horizontal cards, which allow more information to fit alongside the thumbnail than a vertically stacked card on a grid. It also makes each search result more prominent by occupying its own full row, making the reading flow easier.
I'd like to experiment more with a horizontal card layout on the search results. While I believe it's relevant to include this task within the scope of this change, we'd ideally release it as soon as possible and further iteration can be done easily after the feature is out there.
Tasks
Below, we have turned the considerations above into actionable tasks towards completing the initial search implementation:
Current tasks
See Sub-issues below.
Postponed tasks
- Horizontal card layout for search results
- Infinite scrolling
- Search filters
- Text search engine implementation