How we improved Slack retrieval for AI agents by rethinking document boundaries, not just connections.
If you’re building an AI agent that answers questions from Slack data — whether via MCP tools, RAG pipelines, or any other integration — this post is for you.
The problem
When you connect Slack to an AI agent, you expect the agent to understand your team’s conversations. Here’s what Slack data actually looks like when it reaches the model:
- [2026-03-20 14:23] U0847XJKL: :tada: shipped! :rocket:
[2026-03-20 14:24] U0923MXPQ: <@U0847XJKL> :+1:
[2026-03-20 14:25] U0847XJKL: <!channel> FYI migration complete
[2026-03-20 14:30] U1029RLST: @here should we update the docs?
[2026-03-20 14:31] U0847XJKL: yes pls & also update the <link>
:tada: consumes tokens but carries zero information. U0847XJKL can’t be resolved by the model. <!channel> and & are Slack-internal syntax passed through as-is. And the biggest problem: thread discussions get interleaved with unrelated messages, destroying conversational context.
This isn’t a bug in any particular tool or MCP server. Connecting a data source to an LLM is not the same as making that data understandable to an LLM.
Why this matters
Humans naturally ignore emoji reactions, follow threads, and infer meaning from brief messages. LLMs can’t. Every token gets equal weight — :tada: and a critical architecture decision receive the same attention.
Anthropic’s Writing Effective Tools for Agents guide makes the case that tool design directly impacts agent performance. The same principle applies to tool outputs: what the LLM receives matters as much as how it’s connected. A study analyzing 856 tools across 103 MCP servers found that 97.1% had quality issues in their descriptions alone — fixing these improved task success rates by a median of 5.85 percentage points.
The question isn’t “how do we connect Slack?” — that’s solved. The question is what happens to the data between the source and the LLM.
What we did
Three changes to how Slack messages are preprocessed before indexing. Thread splitting had the largest impact. Cleanup and filtering were straightforward engineering, but they compounded.
