Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
Tags

New search backend

A topic by leafo created 12 days ago Views: 432 Replies: 12
Viewing posts 1 to 5
Admin (6 edits) (+15)

Over the past two weeks, we’ve rolled out a new search backend. Refining our search system is an ongoing task, but this latest update represents a significant upgrade in terms of quality and features, so I wanted to highlight some of the notable changes. Keep in mind our search system is separate from our autocomplete system. The search results page is what appears after you type something in the search bar and submit it.

As a reminder, our search system is about finding things by name or title. If you’re looking to browse by category, we recommend using our browse pages and filtering by tag.

Better Text Matching, Higher Performance

The new service we’re using has much better support for various types of queries, including different languages and typo tolerance. It should now match substrings from your query in various new ways to help you find what you’re looking for. Those trying to find your game should have a substantially easier time doing so.

The system is substantially more performant than our old one. For this reason, we’re able to start indexing more fields related to each project page and provide more ways to find and filter pages.

Filtering Added

We’re now indexing the page type (games, assets, tabletop, etc.) with every indexed page. You can filter by these from the top of the search results page. We’ll likely add more filtering options in the future, but for now, this should satisfy the biggest complaint we received about not being able to find a particular type of item by name. For example, searching for Game Assets with “Cave” in their name.

Updated Fields Indexed for Games

We’ve changed what fields from game and project pages are used for indexing. The following is the list of fields that may match a user’s query in order of precedence:

  1. Title (including any title aliases set by a site admin)
  2. Project URL (if different from the title)
  3. Creator’s name and any contributor names
  4. Short description or tagline

The biggest changes here are that all contributor names contribute to a result, and the short description can be used for search terms.

Because we’re now indexing pages by their author names, we’ve temporarily removed the account search functionality from the search results page. We may revisit this in the future.

Jam Search Added

We’re now indexing all jams that are eligible for display. The indexing eligibility is the same as the eligibility for the jam browsing pages: The page must be published, the “Unlisted” option must be unchecked on the jam edit page, and new jam hosts will require human approval for their first jams. Example search, find all the GitHub Game Off jams.

Jam results can be filtered by their status, e.g., in progress, finished, upcoming, etc.

The fields used to index a jam are:

  1. Title
  2. URL (if different from title)
  3. Names of hosts
  4. The short description

Backend Processing

Our new search service runs independently of our primary database. For this reason, we now asynchronously index pages when changes are made by the creators. There may be a small delay between when a page is updated and when and how it appears in search results. Keep this in mind if you’re trying to test your project’s visibility and you just made changes to the page. You may want to check back in an hour or so if you don’t see what you expect.

(+2)

Well done.

Might want to update now the help text inside the search box. "Search for games or creators".

And maybe consider a clarification at some places, that this box will not search tags. There are many threads in community where this is not understood: that there is a second search box that is a write in box with non fixed tags you can enter and filter with, in addition to the suggested tags. It also has misleadingly a help text, that reads: "Select a tag". I remember a heated discussion where someone did not understand that  (View all tags) is simply an untrue statement, if taken literall. It is all "suggested" tags. Not all tags. It is not an exhaustive list. It was months after first using itch that I realized that one can actually write in that drop down box, and not just as a quick select for the entries of that box. As a counterexample, the tags on Steam are exhaustive.

In other words, the search for example with "banana" will give different results than the tag browse page of "banana".

The search box results imply that it will deliver tags, if you happen to search for a suggested tag. Like "horror". It will spell out for you several horror tags. But if you search for banana, it will not notify you, that there is in fact a banana tag used by about 200 projects. Searching for banana will give the usual roundabout 60 projects and people make threads about not understanding, that search results are cut off, or complain why their game is not in the results. A simple note at bottom of search result might deal with that. (Reminding that those results are mostly literal title search and that there is an unlimited tag browsing available as well, and that results were cut off. Especially single word searches might have been an attempt to browse/search for a tag. Or a keyword - something neither search nor tag browsing can achieve, but some people might expect from a box called search.)

Community announcements are all good and well, but the vast majority just does not read here.

If you’re looking to browse by category, we recommend using our browse pages and filtering by tag.

Something like this I would imagine at top or bottom of the search results. Along with the info, that these results were capped and that full titles might be required to find what is searched.

--

Oh, and try using a zoom level greater than 100% or a less than wide screen. The filter_label will flow over. I am surprised you did not use the same drop down box as seen on the browse page, to switch between games and assets.

(+2)

Great job leafo

sounds great.

Thanks for the update! Gave it a spin and I love it~ I love that typo tolerance too! It does help when you can only partially remember the name of something you're looking for.

Quick question: Will the search function have a pagination system, such as a "Next Page" at the bottom of the page for searches with a large amount of results?


I did notice a cutoff after a certain amount is displayed, and there's no Next Page link to click on, meaning I can't browse further than the initial page of results. It's not affected by auto-loading as a setting either. I checked and searching by tags has a pagination system, while searching by keyword (using the search bar) doesn't.

If not, it's fine too. Thanks again! 🙏

(1 edit)
It does help when you can only partially remember the name

That feature existed before. It was improved. But it still is title matching. It now matches url and creator display names too. Short description was matched before, but as far as I observed, if it could fill the results with title matches, those usually had more relevance.

The number of results seems capped more than before. It was something like x = exact title matches + 60 or less. It seems now to be x = 60 or less with exact title matches no longer leading the results, nor being mandatory included.

Also, the relevance ordering seems kinda strange.

If you type neighbor, it will show you the game "That's not my Neighbor" in suggestions starting with "ne" already. But the search

https://itch.io/search?q=neighbor 

does not even have that game in the results (currently). And "neighbor" is the only real noun in that title.

The game will show up for "that's", and even for "that". But it currently does not show up for "thats". So much for the typo correction. It does show up for "thats not my", but since the game also does show up for "not my", that does not tell us anything.

--

The query results have changed since. The game shows now with "ne" or more, as a search term. It will also show up for "thats". 

That the search will match 60 results to "ne" is curious. While this is not a good term to search with, one would expect to match a lot more than 60 titles. But those everyday short words are not good titles. Like fps for example. Those are better used as tags, not as title fragments. You do not search with "fps" to find "banana fps".

In my opinion it depends all on what the search is supposed to do and to make it understood to the people using that feature - without need to consult a message board. I have different expectations from a box called "Title Search" in contrast to "Tag Search", "Keyword Search" or just "Search". Or expectations about a box that does not even tell it is a free write in box and disguises as a drop down list.

Admin(+1)

We probably won’t be adding pagination. Can you give me an example of a query you made where you felt like you wanted to go through multiple pages to find something?

Hi leafo, thanks for letting me know!

Sure, here's the query in question: https://itch.io/search?q=victorian

Admin(+1)

I suppose I was interested more about your use case, not just the query used. Are you looking with something with “Victorian” in the name, or you want to find stuff with that topic?

I was more or less looking for the latter - Victorian-related stuff, which may or may not be in the name or description. For example Victorian-inspired games, assets, comics, etc.

Admin(+2)

I think this is a perfect example of a use case for a tag. I’ve gone ahead and added a new “suggested tag” so that developers can start tagging their work as “Victorian”. The results page will now suggest you browse that tag as well.

Since itch has free tagging, 70+ games already did have that tag.

For single word searches you might want to consider to insert some short blurb anyways, even especially if there are no suggested tags as a result. It could read similar to "Missing something? Maybe try freestyle tag filtering instead (show tag filter box here)". There are many tags that are used and have many items, but are not in the suggested list.

And that tags are freestyle is not really advertised prominently. I remember a heated discussion where I even was called a liar, for claiming that tags are freely choosable and was quoted the then 14 pages of "all" tags.

Thank you so much for the update! I've tried it out and it's great! I hope the developers will start tagging them so they'll be easier to find in the future. Bless~