Skip to main content

Mistral Extremely Slow

Feature Requests

Recently, responses from mistral have been getting very slow when using Mistral (this is the only provider I use). Today, it’s taking several minutes or more for a response to start generating. Once the text starts generating, it goes at a normal speed, but the same delay occurs for each subsequent response. When I run the same prompt in Chatbox on the came computer, there’s no delay. I’m not sure what the problem is. I’m using the most up to date version of BoltAI on MacOS on an Apple M1 MacBook Pro.

Posted by MDNull ·
Status: Closed2 comments

Log in to comment and vote

Comments2

  • Daniel Nguyen

    Team•

    Apr 22

    Can you share the chat with me. Menu Conversation > Export chat. My best guess is the context getting bigger and it affects TTFT (time to first token). BoltAI uses standard Apple network stack so I don’t think the problem was in the networking.

    Also can you try restarting the app and/or restarting wifi?

    • MDNull

      •

      Apr 22

      I think you’re right. If I use a simple prompt with no attachment, it’s quick. But when I create a brand new chat with a fair amount of context in an attched in a file, then it’s very slow (slower than I thought it used to be, but maybe I was mistaken). When I replicated the same new chat with the same attachment in Chatbox, there’s virtually no delay, so I thought it was a problem specific to BoltAI. But I dug into a bit more, and I think Chatbox is truncating the attached file, and that’s why it has no significant delay.

      boltai-chat-d27b4b53-c5c5-4ce0-ab97-d61a553307a9.json

      8.1 KB• Text