feat(agent): Agent tab refinements: one-state composer, no snapping, notifications, choices, subagents, dictation, game frame, copy, History search (#2626)

* fix(agent): one-state composer with model and usage on top; reading back stays detached while the agent streams

- The composer no longer has a compact reading bar that a first tap expands: the editor is
  always the editor, and the model picker and context ring move to the top beside the menu.
- The pet chat drops its DeeDee title so that top bar gets the whole width.
- Scrolling back stops following new output; new entries and streamed text only scroll the
  list while it is at the bottom, and sending a message resumes following.
- When the editor slides back in or the keyboard rises, a list at the bottom keeps its last
  lines above the editor, even mid-drag.

* fix(agent): a revise while reading back shows the editor it focuses

* feat(agent): dictate a message, and attach what the running game shows

- With nothing typed, the editor's send button dictates instead, through Android's speech
  recognizer; what is heard follows the text already there. It asks for the microphone the
  first time.
- While a game session is open, a camera button attaches the game's current frame. It is
  copied from the game's own surface, as the bottom screen's screenshots are, so the chat or
  drawer over it is not in the picture.

* feat(agent): a notification when a turn ends unseen, and replies that end in choices offer them as buttons

- A turn that ends while neither the Agent page nor the pet is in view posts a notification:
  finished, failed or asking, with the reply's first line. Opening the Agent page clears it.
- Claude Code's question tool is not available in print mode, so the brief asks the agent to
  end a reply with a choices block instead. The chat shows its options as buttons, and the one
  picked is sent as the next message.

* feat(agent): a subagent's work shows under the call that started it

Claude Code tags a subagent's messages with the call that started it. Its text, thinking and
streamed deltas used to show as the main agent's reply, and its tool calls mixed into the
turn. Now only its tool calls are kept, stored with their parent, and shown inside the
parent's card: a live line with the step count and the latest call, and every call when
the card is opened. Its token counts no longer overwrite the context reading.

* feat(agent): copy a reply or a tool's text, and open a turn's changes in one place

- A turn's footer has a copy button for everything the agent said in it, and tool input and
  output boxes have one each.
- The files-changed chip opens every edit the turn made, file by file.

* feat(agent): search conversations in History

A field above the list keeps the conversations whose title, messages or replies hold the
text, ignoring case. Chat files are read off the main thread, a quarter second after the
last keystroke.

* fix(agent): sending while reading back jumps to the latest

* feat(agent): manage Claude Code's MCP servers and skills from the Agent menu

A new panel lists the MCP servers claude mcp list reports, adds one for every project from a
URL (over HTTP) or a start command (over stdio), and removes it; connectors from the
claude.ai account are listed but managed there. It also lists the skills in
~/.claude/skills with their descriptions, and deletes one after a confirming second tap.

* fix(agent): review follow-ups for the Agent tab refinements

- Choices only come from one fenced block that ends the reply.
- The Agent page counts as on screen only while its activity is started, so a turn ending
  during a game still notifies.
- Picking a choice or Go jumps to the latest like sending does.
- Dictation stops when the chat is hidden or the keyboard sends; stopping it by hand keeps
  the recognizer until its final result.
- A failed JPEG encode is not attached.
- The copied reply is read from the turn each time.
- MCP start commands keep their quoted arguments; the CLI is read beside its 90 s deadline;
  deleting a skill runs off the main thread and removes links without following them.

* fix(agent): scroll the shrinking list after layout, not during measure

* fix(agent): fence-aware choices parsing; dictation stops overwriting edited drafts

* fix(agent): ignore choices blocks nested in a code fence

* fix(agent): treat tilde fences like backtick fences when finding choices

* test(agent): cover a tilde-fenced choices block
This commit is contained in:
Kurt Himebauch authored and GitHub committed 2026-11-14 17:35:54 -05:00
1 parent a3be5f7ae2
commit eb190f2dd7
49 files changed
+1040 -296

No files matched your search

@@ -1,9 +1,13 @@
package com.droiddeck.launcher.agent
import android.content.Context
import android.graphics.Bitmap
import android.net.Uri
import android.provider.OpenableColumns
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateListOf
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.setValue
import com.droiddeck.launcher.runtime.LinuxRuntime
import java.io.File
@@ -50,6 +54,22 @@ object AgentAttachments {
if (host(context, path).renameTo(moved)) "$GUEST_DIR/$chat/${moved.name}" else path
}
/** Copies the running game's frame; set while a session is open. */
var gameFrame by mutableStateOf<(((Bitmap?) -> Unit) -> Unit)?>(null)
/** Saves [frame] for [chat] (or a new conversation) on the calling thread; returns its guest path. */
fun saveFrame(context: Context, chat: String?, frame: Bitmap): String? = runCatching {
val folder = chat ?: NEW
val dir = File(hostDir(context), folder).apply { mkdirs() }
val target = File(dir, "game-${System.currentTimeMillis()}.jpg")
val written = target.outputStream().use { frame.compress(Bitmap.CompressFormat.JPEG, 88, it) }
if (!written) {
target.delete()
return@runCatching null
}
"$GUEST_DIR/$folder/${target.name}"
}.getOrNull()
/** Deletes copies that will not be sent. */
fun discard(context: Context, files: List<String>) = files.forEach { host(context, it).delete() }
@@ -37,6 +37,21 @@ object AgentChats {
}
.sortedByDescending { it.updated }
/** The chats among [chats] where the user or the agent wrote [query], ignoring case. */
fun search(context: Context, chats: List<Chat>, query: String): List<Chat> {
val q = query.trim()
if (q.isEmpty()) return chats
return chats.filter { chat ->
chat.title.contains(q, ignoreCase = true) || read(file(context, chat.id)).entries.any { matches(it, q) }
}
}
internal fun matches(entry: Entry, query: String) = when (entry) {
is Entry.Asked -> entry.text.contains(query, ignoreCase = true)
is Entry.Said -> entry.text.contains(query, ignoreCase = true)
else -> false
}
fun delete(context: Context, id: String) {
file(context, id).delete()
File(AgentAttachments.hostDir(context), id).deleteRecursively()
@@ -72,6 +87,7 @@ object AgentChats {
is Entry.Thought -> JSONObject().put("thought", entry.text)
is Entry.Result -> JSONObject().put("result", entry.id).put("output", entry.output).put("failed", entry.failed)
is Entry.Used -> JSONObject().put("used", entry.tool).put("detail", entry.detail).put("id", entry.id).put("input", entry.input)
.apply { if (entry.parent.isNotEmpty()) put("parent", entry.parent) }
.apply { entry.output?.let { put("output", it).put("failed", entry.failed) } }
is Entry.Finished -> JSONObject().put("finished", true).put("error", entry.error ?: JSONObject.NULL).put("stopped", entry.stopped).apply {
if (entry.planned) put("planned", true)
@@ -105,7 +121,7 @@ object AgentChats {
o.has("said") -> Entry.Said(o.getString("said"))
o.has("thought") -> Entry.Thought(o.getString("thought"))
o.has("used") -> Entry.Used(o.optString("id"), o.getString("used"), o.optString("detail"), o.optString("input"),
if (o.has("output")) o.optString("output") else null, o.optBoolean("failed"))
if (o.has("output")) o.optString("output") else null, o.optBoolean("failed"), o.optString("parent"))
o.has("result") -> Entry.Result(o.getString("result"), o.optString("output"), o.optBoolean("failed"))
o.has("finished") -> Entry.Finished(
if (o.isNull("error")) null else o.optString("error"), o.optBoolean("stopped"),
@@ -108,8 +108,11 @@ object ClaudeCode : AgentHarness {
override fun parse(line: String): AgentEvent? {
val o = runCatching { JSONObject(line) }.getOrNull() ?: return null
val conversation = o.optString("session_id").takeIf { it.isNotEmpty() }
// A subagent's messages carry the call that started it; only its tool calls are shown, under that call.
val parent = o.optString("parent_tool_use_id").takeIf { it.isNotEmpty() && it != "null" }
return when (o.optString("type")) {
"stream_event" -> {
if (parent != null) return null
val delta = o.optJSONObject("event")?.takeIf { it.optString("type") == "content_block_delta" }?.optJSONObject("delta") ?: return null
when (delta.optString("type")) {
"text_delta" -> AgentEvent(conversation, text = delta.optString("text"))
@@ -123,18 +126,18 @@ object ClaudeCode : AgentHarness {
val entries = (0 until content.length()).mapNotNull { i ->
val block = content.optJSONObject(i) ?: return@mapNotNull null
when (block.optString("type")) {
"text" -> block.optString("text").trim().takeIf { it.isNotEmpty() }?.let { AgentHost.Entry.Said(it) }
"thinking" -> block.optString("thinking").trim().takeIf { it.isNotEmpty() }?.let { AgentHost.Entry.Thought(it) }
"text" -> if (parent != null) null else block.optString("text").trim().takeIf { it.isNotEmpty() }?.let { AgentHost.Entry.Said(it) }
"thinking" -> if (parent != null) null else block.optString("thinking").trim().takeIf { it.isNotEmpty() }?.let { AgentHost.Entry.Thought(it) }
"tool_use" -> {
val input = block.optJSONObject("input")
AgentHost.Entry.Used(block.optString("id"), block.optString("name"), detail(input), fullInput(block.optString("name"), input))
AgentHost.Entry.Used(block.optString("id"), block.optString("name"), detail(input), fullInput(block.optString("name"), input), parent = parent.orEmpty())
}
else -> null
}
}
val usage = message?.optJSONObject("usage")
val context = usage?.let { it.optLong("input_tokens") + it.optLong("cache_read_input_tokens") + it.optLong("cache_creation_input_tokens") + it.optLong("output_tokens") }
AgentEvent(conversation, entries, context = context?.takeIf { it > 0 })
AgentEvent(conversation, entries, context = context?.takeIf { it > 0 && parent == null })
}
"user" -> {
val content = o.optJSONObject("message")?.optJSONArray("content") ?: return conversation?.let { AgentEvent(it) }
@@ -29,8 +29,11 @@ object AgentHost {
data class Asked(val text: String, val files: List<String> = emptyList()) : Entry
data class Said(val text: String) : Entry
data class Thought(val text: String) : Entry
/** A tool call; [output] stays null until its result arrives. */
data class Used(val id: String, val tool: String, val detail: String, val input: String = "", val output: String? = null, val failed: Boolean = false) : Entry
/** A tool call; [output] stays null until its result arrives. [parent] is the subagent call it was made under, if any. */
data class Used(
val id: String, val tool: String, val detail: String, val input: String = "", val output: String? = null, val failed: Boolean = false,
val parent: String = "",
) : Entry
data class Result(val id: String, val output: String, val failed: Boolean) : Entry
/** [planned]: the turn ran in plan mode, so it ends on a plan that nothing has carried out yet. */
data class Finished(val error: String?, val stopped: Boolean = false, val usage: Usage? = null, val planned: Boolean = false) : Entry
@@ -71,7 +74,9 @@ object AgentHost {
"Run it with run_in_background and keep checking its output, so the user can talk to you meanwhile. " +
"The restart ends this process, and this conversation continues on the new build. " +
"Files the user attaches are listed at the end of their message, under /root/.cache/droiddeck/attachments. " +
"Commits and pull requests you make are the user's own: add no co-author line, AI credit or generated-by note."
"Commits and pull requests you make are the user's own: add no co-author line, AI credit or generated-by note. " +
"To have the user pick between options, end your reply with a fenced code block whose language is choices, one short option per line, " +
"and end your turn: DroidDeck shows them as buttons and sends the one picked as the next message."
val entries = mutableStateListOf<Entry>()
val queue = mutableStateListOf<Queued>()
@@ -142,6 +147,7 @@ object AgentHost {
fun seen() {
unseen = false
appContext?.let { AgentNotice.clear(it) }
}
fun refresh(context: Context) {
@@ -462,7 +468,19 @@ object AgentHost {
if (!stopped && queue.isNotEmpty()) {
val next = queue.removeAt(0)
send(app, next.text, next.files)
} else if (!stopped) notifyEnded(app, error)
}
private fun notifyEnded(app: Context, error: String?) {
val start = entries.indexOfLast { it is Entry.Asked }
val said = entries.subList(start + 1, entries.size).lastOrNull { it is Entry.Said } as? Entry.Said
val reply = said?.let { choices(it.text) }
val kind = when {
error != null -> AgentNotice.Kind.FAILED
!reply?.options.isNullOrEmpty() -> AgentNotice.Kind.QUESTION
else -> AgentNotice.Kind.DONE
}
AgentNotice.post(app, kind, error ?: reply?.text?.takeIf { it.isNotBlank() }?.let { firstLine(it) } ?: "")
}
/** A message typed or retried by the user replaces the automatic resend. */
@@ -501,7 +519,7 @@ object AgentHost {
}
/** The latest todo list the agent wrote in this conversation. */
fun todos(): String? = entries.lastOrNull { it is Entry.Used && it.tool == "TodoWrite" }?.let { (it as Entry.Used).input }
fun todos(): String? = entries.lastOrNull { it is Entry.Used && it.tool == "TodoWrite" && it.parent.isEmpty() }?.let { (it as Entry.Used).input }
private fun tick() {
now = SystemClock.elapsedRealtime()
@@ -645,3 +663,34 @@ object AgentHost {
internal fun prefs(context: Context) = context.getSharedPreferences("agent", Context.MODE_PRIVATE)
}
/** A reply split into its text and the options of a trailing choices block. */
internal data class Choices(val text: String, val options: List<String>)
private val FENCE = Regex("""[ \t]{0,3}(`{3,}|~{3,})(.*)""")
/** Walks Markdown fences so a choices block counts only at the top level and only as the last thing in the reply. */
internal fun choices(reply: String): Choices {
val lines = reply.split('\n')
var open = ""
var opener = -1
var openChoices = false
var block: IntRange? = null
lines.forEachIndexed { i, line ->
val fence = FENCE.matchEntire(line)
if (open.isEmpty()) {
if (fence != null) {
open = fence.groupValues[1]
opener = i
openChoices = fence.groupValues[2].trim() == "choices"
block = null
} else if (line.isNotBlank()) block = null
} else if (fence != null && fence.groupValues[1][0] == open[0] && fence.groupValues[1].length >= open.length && fence.groupValues[2].isBlank()) {
block = if (openChoices) opener..i else null
open = ""
}
}
val found = block?.takeIf { open.isEmpty() } ?: return Choices(reply, emptyList())
val options = lines.subList(found.first + 1, found.last).map { it.trim().removePrefix("- ").removePrefix("* ").trim() }.filter { it.isNotEmpty() }
return Choices(lines.subList(0, found.first).joinToString("\n").trimEnd(), options)
}
@@ -0,0 +1,58 @@
package com.droiddeck.launcher.agent
import android.app.ActivityManager
import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.PendingIntent
import android.content.Context
import android.content.Intent
import com.droiddeck.launcher.FrontEndActivity
import com.droiddeck.launcher.R
import com.droiddeck.launcher.ui.AgentPet
/** Tells the user a turn has ended when nothing on screen already does. */
internal object AgentNotice {
private const val CHANNEL_ID = "agent_done"
private const val NOTIFICATION_ID = 4108
enum class Kind { DONE, FAILED, QUESTION }
fun post(context: Context, kind: Kind, text: String) {
if (onScreen(context)) return
val manager = context.getSystemService(NotificationManager::class.java) ?: return
manager.createNotificationChannel(NotificationChannel(CHANNEL_ID, context.getString(R.string.agent_done_channel), NotificationManager.IMPORTANCE_DEFAULT))
val open = PendingIntent.getActivity(context, 2,
Intent(context, FrontEndActivity::class.java)
.setFlags(Intent.FLAG_ACTIVITY_SINGLE_TOP or Intent.FLAG_ACTIVITY_REORDER_TO_FRONT)
.putExtra(com.droiddeck.launcher.stores.download.StoreDownloadService.EXTRA_NAV, "agent"),
PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT)
val title = context.getString(when (kind) {
Kind.DONE -> R.string.agent_done
Kind.FAILED -> R.string.agent_done_failed
Kind.QUESTION -> R.string.agent_done_question
})
runCatching {
manager.notify(NOTIFICATION_ID, Notification.Builder(context, CHANNEL_ID)
.setSmallIcon(R.drawable.ic_stat_session)
.setContentTitle(title)
.setContentText(text)
.setStyle(Notification.BigTextStyle().bigText(text))
.setContentIntent(open)
.setAutoCancel(true)
.setCategory(Notification.CATEGORY_STATUS)
.build())
}
}
fun clear(context: Context) {
context.getSystemService(NotificationManager::class.java)?.cancel(NOTIFICATION_ID)
}
/** The chat in a DroidDeck window in front, or the pet, which shows the reply in its own bubble. */
private fun onScreen(context: Context): Boolean {
if (AgentPet.floating) return true
val state = ActivityManager.RunningAppProcessInfo().also { ActivityManager.getMyMemoryState(it) }
return AgentPet.agentOnScreen > 0 && state.importance <= ActivityManager.RunningAppProcessInfo.IMPORTANCE_FOREGROUND
}
}
@@ -0,0 +1,143 @@
package com.droiddeck.launcher.agent
import android.content.Context
import com.droiddeck.launcher.runtime.LinuxRuntime
import java.io.File
import java.nio.file.FileVisitResult
import java.nio.file.Files
import java.nio.file.LinkOption
import java.nio.file.Path
import java.nio.file.SimpleFileVisitor
import java.nio.file.attribute.BasicFileAttributes
import java.util.concurrent.TimeUnit
/** Claude Code's MCP servers and skills in the userland, through its own CLI and skills folder. */
internal object AgentTools {
data class Server(val name: String, val target: String, val status: String) {
/** Connectors come from the claude.ai account and are managed there, not by the CLI. */
val account get() = name.startsWith("claude.ai ")
}
data class Skill(val name: String, val description: String)
class Result(val ok: Boolean, val output: String)
private const val SKILLS = "root/.claude/skills"
private val SERVER = Regex("""^(.+?): (.+) - (.+)$""")
fun servers(context: Context): List<Server>? {
val result = run(context, listOf(ClaudeCode.binary, "mcp", "list"))
if (!result.ok) return null
return parseServers(result.output)
}
internal fun parseServers(output: String): List<Server> = output.lines().mapNotNull { line ->
SERVER.find(line.trim())?.destructured?.let { (name, target, status) -> Server(name, target, status) }
}
/** A URL is added as an HTTP server; anything else is the command that starts a stdio server. */
internal fun addCommand(name: String, target: String): List<String> {
val t = target.trim()
val base = listOf(ClaudeCode.binary, "mcp", "add", "--scope", "user")
return if (t.startsWith("http://") || t.startsWith("https://")) base + listOf("--transport", "http", name.trim(), t)
else base + listOf(name.trim(), "--") + words(t)
}
/** Splits a command line the way a shell would for quotes and backslashes, without expanding anything. */
internal fun words(line: String): List<String> {
val words = mutableListOf<String>()
val word = StringBuilder()
var inWord = false
var quote: Char? = null
var i = 0
while (i < line.length) {
val c = line[i]
when {
quote == '\'' -> if (c == '\'') quote = null else word.append(c)
c == '\\' && i + 1 < line.length && (quote == null || line[i + 1] == '"' || line[i + 1] == '\\') -> {
word.append(line[++i])
inWord = true
}
quote == '"' -> if (c == '"') quote = null else word.append(c)
c == '\'' || c == '"' -> {
quote = c
inWord = true
}
c.isWhitespace() -> if (inWord) {
words += word.toString()
word.clear()
inWord = false
}
else -> {
word.append(c)
inWord = true
}
}
i++
}
if (inWord) words += word.toString()
return words
}
fun add(context: Context, name: String, target: String) = run(context, addCommand(name, target))
fun remove(context: Context, name: String) = run(context, listOf(ClaudeCode.binary, "mcp", "remove", name))
fun skills(context: Context): List<Skill> =
File(LinuxRuntime.rootDir(context), SKILLS).listFiles { f -> f.isDirectory }.orEmpty().sortedBy { it.name.lowercase() }.map { dir ->
Skill(dir.name, description(File(dir, "SKILL.md")))
}
internal fun description(file: File): String = runCatching {
val lines = file.readLines()
if (lines.firstOrNull()?.trim() != "---") return@runCatching ""
lines.drop(1).takeWhile { it.trim() != "---" }.firstOrNull { it.startsWith("description:") }
?.removePrefix("description:")?.trim()?.trim('"', '\'').orEmpty()
}.getOrDefault("")
fun deleteSkill(context: Context, name: String): Boolean {
if (name.isEmpty() || name.contains('/') || name.startsWith(".")) return false
val root = File(LinuxRuntime.rootDir(context), SKILLS)
val dir = File(root, name)
if (dir.parentFile?.path != root.path) return false
return deleteTree(dir.toPath())
}
/** Deletes [path] and what is under it; a symbolic link is removed itself, never what it points to. */
internal fun deleteTree(path: Path): Boolean = runCatching {
if (!Files.exists(path, LinkOption.NOFOLLOW_LINKS)) return@runCatching true
Files.walkFileTree(path, object : SimpleFileVisitor<Path>() {
override fun visitFile(file: Path, attrs: BasicFileAttributes): FileVisitResult {
Files.delete(file)
return FileVisitResult.CONTINUE
}
override fun postVisitDirectory(dir: Path, exc: java.io.IOException?): FileVisitResult {
if (exc != null) throw exc
Files.delete(dir)
return FileVisitResult.CONTINUE
}
})
true
}.getOrDefault(false)
private fun run(context: Context, command: List<String>): Result {
val child = runCatching { AgentHost.guest(context.applicationContext, command, connectAdb = false).start() }
.getOrElse { return Result(false, it.message.orEmpty()) }
val output = StringBuffer()
// The CLI can hold its output open past the deadline, so it is read beside the wait rather than before it.
val reader = Thread({ runCatching { child.inputStream.bufferedReader().forEachLine { output.append(it).append('\n') } } }, "agent-tools").apply { start() }
return try {
val ended = child.waitFor(90, TimeUnit.SECONDS)
if (!ended) child.destroyForcibly()
reader.join(2_000)
Result(ended && child.exitValue() == 0, output.toString().trim())
} catch (e: InterruptedException) {
child.destroyForcibly()
Thread.currentThread().interrupt()
Result(false, output.toString().trim())
} finally {
runCatching { child.inputStream.close() }
}
}
}