itch.io: browse the Store tab by sort and price, with paging and a disk cache (#1473)

## What this does

The itch.io Store tab used to show only four fixed shelves. This is the
first of a series of PRs that rework it. It adds real browsing: pick an
order and a price filter, and the tab loads itch's own listing for that
choice and keeps loading as you scroll. Login, the library, downloads
and installs are unchanged.

Platform and tag filters, the game page with the install picker, updates
and collections are not in this PR. They come in the following ones.

## How it works

itch serves every listing page as an RSS feed if you add `.xml` to the
address. The app already reads those feeds for the shelves, so browsing
reuses the same parser (`ItchShelves.parse`). Nothing here needs the API
key.

The address is built the way itch builds its own links: sort, then
price, then `platform-<name>`, then `tag-<slug>`.

| Choice | Address |
|---|---|
| Popular (default) | `https://itch.io/games` |
| New & popular | `/games/new-and-popular` |
| Top sellers | `/games/top-sellers` |
| Top rated | `/games/top-rated` |
| Most recent | `/games/newest` |
| Free | `/games/free` |
| On sale | `/games/on-sale` |
| Paid | `/games/store` |
| $5 or less | `/games/5-dollars-or-less` |
| $15 or less | `/games/15-dollars-or-less` |

Choices combine, for example `/games/top-rated/free`, then `.xml` for
the feed. Page 2 and later add `?page=N`.

## Changes

**`stores/itch/ItchBrowse.kt` (new)**
- `BrowseQuery` holds sort, price, platforms and tags. The default query
is the front page, which keeps the existing shelves.
- `ItchBrowse.path` and `url` build the addresses above. Only the
spellings listed above are used. Platforms and tags are already accepted
by the builder, ready for the next PR.
- `ItchBrowse.page` loads one page of results:
- Disk cache under `cacheDir/itch-browse`, keyed by the SHA-1 of the
address, fresh for 3 hours.
  - Requests are made one at a time, at least 1.2 s apart.
  - A 429 answer waits 8 s and tries once more.
- Each request is written to the store log (address, status, number of
games).
- `ItchBrowseState` is the Compose state for the tab: the query, the
games loaded so far, and the loading, ended and failed flags. A
generation counter discards answers that arrive after the user has
changed the query. Paging skips games it has already shown.

**`ui/ItchBrowseBar.kt` (new)**
- A sort chip row and a price chip row, plus a load-more trigger at the
end of the grid.
- Chips are selectable, work with the controller (`controllerConfirm`)
and use the same glide border as the rest of the app.

**`ui/StoresPage.kt`**
- The storefront shows the chip bar for itch.io and, when a non-default
choice is active, the browse results in the existing grid instead of the
shelves.
- Games you already own are marked as owned in the results.
- Empty and failed states have their own messages.

**`stores/itch/ItchShelves.kt`**
- `open(url)` is now `internal` so browsing can reuse it.

**`stores/StoresState.kt`**
- `detailItem` also looks in the browse results for itch.io, so tapping
a result opens its detail page.

**Strings**
- 13 new strings (sort names, price names, "Load more", "Couldn't load
games"), translated in all 33 locales.

**Docs and tests**
- `docs/development/itch-browse.md` explains the address scheme, the
cache and the paging.
- `ItchBrowseTest` checks the addresses the code builds against the ones
listed above.

## Testing

I checked every address with `.xml` added, and all of them load without
errors:
- Every sort, every price filter and every platform on its own.
- Combinations: `top-rated/free`, `top-rated/on-sale`,
`top-rated/store`, `top-rated/5-dollars-or-less`,
`top-rated/free/platform-windows` and
`top-rated/free/platform-windows/tag-pixel-art`.
- Paging: `?page=2` and `top-rated.xml?page=2`.

The unit test locks those addresses in.

## Not in this PR

- Platform and tag filters in the UI, and search.
- The game page, the version picker and install for free games.
- Updates for installed games and itch collections.
- Android builds will link out to the itch.io page. Butler patching
comes last, as an add-on pack.
This commit is contained in:
MaxsTechReview authored and GitHub committed 2026-10-24 11:33:47 -04:00
commit 77e23cb397
40 files changed
+840 -8

No files matched your search

@@ -126,7 +126,9 @@ object StoresState {
fun installedGame(store: Store, id: String): InstalledStoreGame? = installed.firstOrNull { it.sidecar.store == store && it.sidecar.id == id }
fun detailItem(store: Store, id: String): CatalogItem? = detailCatalogItem(
store, id, library[store].orEmpty(), found[store].orEmpty(), shelves[store],
store, id, library[store].orEmpty(),
found[store].orEmpty() + if (store == Store.ITCH) com.droiddeck.launcher.stores.itch.ItchBrowseState.items else emptyList(),
shelves[store],
)
fun open(context: Context, store: Store, force: Boolean = false) {
@@ -0,0 +1,176 @@
package com.droiddeck.launcher.stores.itch
import android.content.Context
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.setValue
import com.droiddeck.launcher.stores.CatalogItem
import com.droiddeck.launcher.stores.StoresState
import org.json.JSONArray
import java.io.File
import java.security.MessageDigest
internal data class BrowseQuery(
val sort: String = ItchBrowse.SORT_POPULAR,
val price: String = ItchBrowse.PRICE_ANY,
val platforms: Set<String> = emptySet(),
val tags: List<String> = emptyList(),
) {
val isDefault: Boolean get() = this == BrowseQuery()
}
internal class BrowsePage(val items: List<CatalogItem>, val code: Int, val cached: Boolean)
internal object ItchBrowse {
const val SORT_POPULAR = "popular"
const val SORT_NEW_POPULAR = "new-and-popular"
const val SORT_TOP_SELLERS = "top-sellers"
const val SORT_TOP_RATED = "top-rated"
const val SORT_NEWEST = "newest"
val sorts = listOf(SORT_POPULAR, SORT_NEW_POPULAR, SORT_TOP_SELLERS, SORT_TOP_RATED, SORT_NEWEST)
const val PRICE_ANY = "any"
const val PRICE_FREE = "free"
const val PRICE_SALE = "on-sale"
const val PRICE_PAID = "store"
const val PRICE_UNDER_5 = "5-dollars-or-less"
const val PRICE_UNDER_15 = "15-dollars-or-less"
val prices = listOf(PRICE_ANY, PRICE_FREE, PRICE_SALE, PRICE_PAID, PRICE_UNDER_5, PRICE_UNDER_15)
val platformNames = listOf("windows", "osx", "linux", "android", "ios", "web", "mobile-web")
private const val BASE = "https://itch.io/games"
private const val FRESH_MS = 3 * 60 * 60 * 1000L
private const val GAP_MS = 1_200L
private const val RETRY_MS = 8_000L
private const val FOLDER = "itch-browse"
@Volatile private var lastRequestAt = 0L
private val lock = Any()
private fun sortSegment(sort: String): String? = sort.takeIf { it != SORT_POPULAR && it in sorts }
private fun priceSegment(price: String): String? = price.takeIf { it != PRICE_ANY && it in prices }
fun path(query: BrowseQuery): String {
val segments = listOfNotNull(sortSegment(query.sort), priceSegment(query.price)) +
query.platforms.sorted().map { "platform-$it" } +
query.tags.map { "tag-" + tagSlug(it) }
return if (segments.isEmpty()) BASE else BASE + "/" + segments.joinToString("/")
}
fun url(query: BrowseQuery, page: Int): String = path(query) + ".xml" + if (page > 1) "?page=$page" else ""
fun tagSlug(tag: String): String = tag.trim().lowercase().replace(Regex("[^a-z0-9]+"), "-").trim('-')
fun page(context: Context, query: BrowseQuery, page: Int, force: Boolean = false): BrowsePage {
val app = context.applicationContext
val url = url(query, page)
if (!force) read(app, url)?.let { return BrowsePage(it, 200, true) }
val (status, body) = fetch(url)
val items = if (status in 200..299 && body != null) ItchShelves.parse(body) else emptyList()
StoresState.logLine("itch.io browse: $url answered $status with ${items.size} games")
if (items.isNotEmpty()) write(app, url, items)
return BrowsePage(items, status, false)
}
private fun fetch(url: String): Pair<Int, String?> {
synchronized(lock) {
pause()
var answer = ItchShelves.open(url)
if (answer.first == 429) {
StoresState.logLine("itch.io browse: 429, waiting before one more try")
Thread.sleep(RETRY_MS)
pause()
answer = ItchShelves.open(url)
}
lastRequestAt = System.currentTimeMillis()
return answer
}
}
private fun pause() {
val gap = GAP_MS - (System.currentTimeMillis() - lastRequestAt)
if (gap > 0) Thread.sleep(gap)
}
private fun file(context: Context, url: String): File {
val digest = MessageDigest.getInstance("SHA-1").digest(url.toByteArray()).joinToString("") { "%02x".format(it) }
return File(File(context.cacheDir, FOLDER), "$digest.json")
}
private fun read(context: Context, url: String): List<CatalogItem>? {
val file = file(context, url)
if (!file.isFile || System.currentTimeMillis() - file.lastModified() > FRESH_MS) return null
return runCatching { ItchCatalog.list(JSONArray(file.readText())) }.getOrNull()?.takeIf { it.isNotEmpty() }
}
private fun write(context: Context, url: String, items: List<CatalogItem>) {
val file = file(context, url)
runCatching {
file.parentFile?.mkdirs()
file.writeText(ItchCatalog.array(items).toString())
}
}
}
internal object ItchBrowseState {
var query by mutableStateOf(BrowseQuery())
private set
var items by mutableStateOf<List<CatalogItem>>(emptyList())
private set
var loading by mutableStateOf(false)
private set
var ended by mutableStateOf(false)
private set
var failed by mutableStateOf(false)
private set
private var next = 1
private var generation = 0
val active: Boolean get() = !query.isDefault
fun choose(context: Context, wanted: BrowseQuery) {
if (wanted == query) return
generation++
query = wanted
items = emptyList()
ended = false
failed = false
loading = false
next = 1
if (active) more(context)
}
fun more(context: Context, retry: Boolean = false) {
if (loading || ended || !active || (failed && !retry)) return
loading = true
failed = false
val app = context.applicationContext
val mine = generation
val asked = query
val page = next
Thread({
val answer = runCatching { ItchBrowse.page(app, asked, page) }.getOrNull()
StoresState.post {
if (mine != generation) return@post
loading = false
if (answer == null || answer.code !in 200..299) {
failed = true
return@post
}
if (answer.items.isEmpty()) {
ended = true
return@post
}
val known = items.mapTo(HashSet()) { it.id }
val fresh = answer.items.filter { known.add(it.id) }
if (fresh.isEmpty()) ended = true else {
items = items + fresh
next = page + 1
}
}
}, "itch-browse").start()
}
}
@@ -108,7 +108,7 @@ internal object ItchShelves {
return shelves.takeUnless { it.isEmpty }
}
private fun open(url: String): Pair<Int, String?> {
internal fun open(url: String): Pair<Int, String?> {
var conn: HttpURLConnection? = null
return try {
conn = (URL(url).openConnection() as HttpURLConnection).apply {
@@ -0,0 +1,112 @@
package com.droiddeck.launcher.ui
import androidx.compose.foundation.LocalIndication
import androidx.compose.foundation.background
import androidx.compose.foundation.horizontalScroll
import androidx.compose.foundation.hoverable
import androidx.compose.foundation.interaction.MutableInteractionSource
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.selection.selectable
import androidx.compose.foundation.selection.selectableGroup
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.LinearProgressIndicator
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.semantics.Role
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import com.droiddeck.launcher.R
import com.droiddeck.launcher.stores.itch.ItchBrowse
import com.droiddeck.launcher.stores.itch.ItchBrowseState
@Composable
internal fun ItchBrowseBar() {
val ctx = LocalContext.current
val query = ItchBrowseState.query
Column(Modifier.fillMaxWidth().padding(top = 6.dp, bottom = 8.dp), verticalArrangement = Arrangement.spacedBy(6.dp)) {
ChipRow {
for (sort in ItchBrowse.sorts) BrowseChip(sortLabel(sort), query.sort == sort) { ItchBrowseState.choose(ctx, query.copy(sort = sort)) }
}
ChipRow {
for (price in ItchBrowse.prices) BrowseChip(priceLabel(price), query.price == price) { ItchBrowseState.choose(ctx, query.copy(price = price)) }
}
}
}
@Composable
internal fun ItchLoadMore() {
val ctx = LocalContext.current
val count = ItchBrowseState.items.size
LaunchedEffect(Unit) { ItchBrowseState.more(ctx) }
if (ItchBrowseState.loading) LinearProgressIndicator(Modifier.fillMaxWidth().padding(vertical = 12.dp))
else if (ItchBrowseState.failed) Box(Modifier.fillMaxWidth().padding(vertical = 8.dp), contentAlignment = Alignment.Center) {
BrowseChip(stringResource(R.string.stores_dl_retry), false) { ItchBrowseState.more(ctx, retry = true) }
}
else if (!ItchBrowseState.ended && count > 0) Box(Modifier.fillMaxWidth().padding(vertical = 8.dp), contentAlignment = Alignment.Center) {
BrowseChip(stringResource(R.string.stores_itch_load_more), false) { ItchBrowseState.more(ctx) }
}
}
@Composable
private fun ChipRow(content: @Composable () -> Unit) {
Row(
horizontalArrangement = Arrangement.spacedBy(6.dp), verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth().horizontalScroll(rememberScrollState()).selectableGroup(),
) { content() }
}
@Composable
private fun BrowseChip(label: String, selected: Boolean, onClick: () -> Unit) {
val colors = MaterialTheme.colorScheme
val pal = LocalPalette.current
val shape = RoundedCornerShape(8.dp)
val src = remember { MutableInteractionSource() }
val hot = rememberHot(src)
Text(
label, fontSize = 13.sp, fontWeight = FontWeight.SemiBold, maxLines = 1,
color = if (selected) pal.onSignal else if (hot) colors.onBackground else colors.onSurfaceVariant,
modifier = Modifier.clip(shape).background(if (selected) pal.signal else colors.surfaceVariant)
.glideBorder(hot, shape, pal.signal)
.hoverable(src).selectable(selected = selected, interactionSource = src, indication = LocalIndication.current, role = Role.Tab, onClick = onClick)
.controllerConfirm(onClick = onClick)
.padding(horizontal = 12.dp, vertical = 7.dp),
)
}
@Composable
private fun sortLabel(sort: String): String = stringResource(
when (sort) {
ItchBrowse.SORT_NEW_POPULAR -> R.string.stores_itch_sort_new_popular
ItchBrowse.SORT_TOP_SELLERS -> R.string.stores_itch_sort_top_sellers
ItchBrowse.SORT_TOP_RATED -> R.string.stores_itch_sort_top_rated
ItchBrowse.SORT_NEWEST -> R.string.stores_itch_sort_new
else -> R.string.stores_itch_sort_popular
},
)
@Composable
private fun priceLabel(price: String): String = stringResource(
when (price) {
ItchBrowse.PRICE_FREE -> R.string.stores_itch_price_free
ItchBrowse.PRICE_SALE -> R.string.stores_itch_price_sale
ItchBrowse.PRICE_PAID -> R.string.stores_itch_price_paid
ItchBrowse.PRICE_UNDER_5 -> R.string.stores_itch_price_under_5
ItchBrowse.PRICE_UNDER_15 -> R.string.stores_itch_price_under_15
else -> R.string.stores_itch_price_any
},
)
@@ -146,6 +146,7 @@ import com.droiddeck.launcher.R
import com.droiddeck.launcher.frontend.Library
import com.droiddeck.launcher.session.SessionPrefs
import com.droiddeck.launcher.stores.CatalogItem
import com.droiddeck.launcher.stores.itch.ItchBrowseState
import com.droiddeck.launcher.stores.Store
import com.droiddeck.launcher.stores.StoreTab
import com.droiddeck.launcher.stores.StoresState
@@ -961,7 +962,8 @@ private fun Storefront(
val everything = remember(library, shelves, found) { (found + library + (shelves?.all ?: emptyList())).distinctBy { it.id } }
LaunchedEffect(store, q) { StoresState.search(ctx, store, q) }
val loaded = StoresState.library.containsKey(store)
val showGrid = tab != "store" || q.isNotEmpty()
val browsing = store == Store.ITCH && tab == "store" && q.isEmpty() && ItchBrowseState.active
val showGrid = tab != "store" || q.isNotEmpty() || browsing
val sort = s.storesSort
val lastPlayedFor = remember(s.steamGames) {
val byKey = s.steamGames.associate { "${it.source}:${it.storeId}" to it.lastPlayed }
@@ -974,8 +976,13 @@ private fun Storefront(
lookup
}
val isSizeSort = tab == "library" && (sort == "sizeAsc" || sort == "sizeDesc")
val candidates = remember(library, everything, tab, q, s.storesShowMature) {
if (tab == "library") library.filter(::matches) else everything.filter { matches(it) && shown(it) }
val browseItems = if (browsing) ItchBrowseState.items else emptyList()
val candidates = remember(library, everything, tab, q, s.storesShowMature, browsing, browseItems) {
if (tab == "library") library.filter(::matches)
else if (browsing) {
val owned = library.associateBy { it.id }
browseItems.map { owned[it.id] ?: it }.filter(::shown)
} else everything.filter { matches(it) && shown(it) }
}
val candidateKeys = remember(candidates) { candidates.map { it.key } }
var resolvedSizeSortKey by remember(store, tab) { mutableStateOf<List<String>?>(null) }
@@ -1013,6 +1020,8 @@ private fun Storefront(
}
val empty = stringResource(when {
tab == "library" && library.isEmpty() && StoresState.status[store] != null -> R.string.stores_library_loading
browsing && ItchBrowseState.loading -> R.string.stores_shelves_loading
browsing && ItchBrowseState.failed -> R.string.stores_itch_browse_failed
else -> R.string.stores_nothing_matches
})
var firstPlaced = false
@@ -1023,8 +1032,9 @@ private fun Storefront(
// A bar only where the tab has nothing else on it: shelves, the shelf of owned games, or a grid still loading.
val gridEmpty = if (showGrid) visible.isEmpty() else (shelves == null || shelves.isEmpty) && library.isEmpty()
val barLine = StoresState.status[store]?.takeIf { gridEmpty && syncCount(it) != null }
val itchBar = store == Store.ITCH && tab == "store" && q.isEmpty()
val headerItems = 1 + (if (sizeSortPending) 1 else 0) + (if (barLine != null) 1 else 0) +
(if (StoresState.problems[store] != null) 1 else 0)
(if (StoresState.problems[store] != null) 1 else 0) + (if (itchBar) 1 else 0)
val listMode = s.storesView == "list" && tab == "library"
val capsules = s.storesView == "capsules" && tab == "library"
val tileWidth = if (capsules) CapsuleWidth else CardWidth
@@ -1110,6 +1120,7 @@ private fun Storefront(
}
barLine?.let { line -> item(key = "sync", span = { GridItemSpan(maxLineSpan) }) { Rise(3) { SyncBar(line) } } }
StoresState.problems[store]?.let { line -> item(key = "problem", span = { GridItemSpan(maxLineSpan) }) { Rise(3) { Box(Modifier.padding(bottom = 8.dp)) { Note(line) } } } }
if (itchBar) item(key = "itch-browse-bar", span = { GridItemSpan(maxLineSpan) }) { ItchBrowseBar() }
when (tab) {
"library" -> if (!loaded) Unit
else if (split == 0) storeGrid(visible, empty, card)
@@ -1120,8 +1131,10 @@ private fun Storefront(
storeGrid(visible.drop(split), empty, card)
}
else -> {
if (q.isNotEmpty()) storeGrid(visible, empty, card)
else {
if (q.isNotEmpty() || browsing) {
storeGrid(visible, empty, card)
if (browsing) item(key = "itch-more", span = { GridItemSpan(maxLineSpan) }) { ItchLoadMore() }
} else {
item(key = "shelves", span = { GridItemSpan(maxLineSpan) }) {
Column {
Shelf(stringResource(R.string.stores_shelf_new), shelf(shelves?.whatsNew), card)