Skip to main content

Major increase in input tokens after the latest update – long-context chats are becoming unusable

Bug Reports

Hello Daniel

I'm a writer and researcher, and I use BoltAI extensively for working on long-running writing projects.

My workflow depends heavily on maintaining a single, very long conversation containing the entire history of a project: details, timelines, research, notes, previous discussions and decisions. I also use a substantial system prompt.

I deliberately use Context Policy: All messages because I need the AI to have access to the entire conversation, from the very beginning. This is fundamental to how I work.

The problem started after the latest update.

I haven't changed my context settings. I still use All messages, and I'm working in the same existing conversation.

However, the number of input tokens being sent to the model has increased dramatically.

Here are two examples using the same model (GPT-6 Luna) and the same long-running chat:

BEFORE THE UPDATE – November 3

  • Input tokens: 132,670

  • Cached tokens: 100,735

  • Output tokens: 965

  • Estimated cost: $0.004683

AFTER THE UPDATE – November 9

  • Input tokens: 799,066

  • Output tokens: 821

  • Estimated cost: $0.0803

That's approximately 6 times more input tokens for what should be essentially the same conversation context.

I also encountered this OpenRouter error:

"The endpoint's maximum context length is 1,048,576 tokens. However, you requested about 1,341,308 tokens (1,209,993 of text input, 243 of tool input, 131,072 in the output)."

This means that some models that previously worked with my conversations can no longer process them at all. Others can technically handle the request, but the amount of input tokens and the cost have increased enormously.

I noticed the recent changelog mentions improvements to how chat history and attachments are sent to models. I suspect something in the way BoltAI now builds, serializes or includes the conversation history has changed.

My questions:

  1. Has the way BoltAI constructs and sends conversation history changed in the latest update? Could this explain the significant increase in input tokens?

  2. Is there a way to restore the previous behaviour or configure how the conversation history is serialized and sent?

  3. Is there a way to reduce the token overhead without losing the ability to give the AI access to the entire conversation and the system prompt?

What I really need is to preserve the full conversational context while avoiding this massive increase in input tokens.

I absolutely rely on BoltAI for this workflow, and until this update it worked extremely well. Unfortunately, the current behaviour makes it increasingly difficult and expensive to continue using it normally.

Could you clarify what changed and whether there is a way to address this?

Thank you!

Posted by MubiClarke ·
Status: In QA2 comments

Log in to comment and vote

Comments2

  • Daniel Nguyen

    Team•

    Nov 10

    Hi. I’ve released v3.1.0 to address this issue. I added the “Auto-compact” option where BoltAI would “compact” the entire chat when it reaches the Compact Threshold (default 32K tokens - configurable).

    You can also compact it manually with the slash command /compact

    To check the current context, click the ring icon next to the mic (see screenshot)

    Give it a try and let me know if it works for you.

  • Daniel Nguyen

    Team•

    Nov 10

    Hi,

    Thanks for the detailed numbers, they made this easy to track down.

    Yes, this changed in 2.18.5. Before that update, a bug meant "All messages" only sent the most recent messages loaded on screen, roughly the last 20, and not the whole chat. So your requests on Nov 3 never included the full history, even though the setting said they did. 2.18.5 fixed that, and BoltAI now sends every message in the chat. That's where the jump to ~800k tokens comes from.

    The important part: every model has a fixed context window, the most text it can read in one request. For the model in your error, that's 1,048,576 tokens. It's a hard limit set by the model, not by BoltAI, and no app can send more than that. Your chat with its attachments is now about 1.2M tokens, so it no longer fits in one request, whatever the settings. The chat will keep growing, so even models with bigger windows will hit their limit eventually.

    So keeping the whole project in one chat and sending all of it every time can't keep working long-term. These options keep the important context and stay under the limit:

    • Last N pairs (up to 50) sends only recent history. Put the parts the model must always know (timelines, key decisions, research notes) in the system prompt or in project files, which go with every message whatever this setting is.

    • Start a new chat per phase of the project. Paste a summary of the earlier chat in at the start.

    • Lower Max Tokens. Your error shows 131,072 tokens reserved for the reply, and that counts against the same window. A few thousand is plenty for most replies.

    • Old attachments are re-sent with every message. If early messages carry large files, moving them to project files cuts a lot.

    The help article explains each Context Policy option and the context limit: https://help.boltai.com/en/articles/5896489-chat-parameters-in-the-inspector

    Cheers