Why does ChatGPT give different answers to the same question?

ChatGPT gives different answers to the same question because it does not look answers up. It writes them, one small piece at a time, and at each step it picks from several likely options with a measure of chance. That alone is enough to change the wording, the order and sometimes the substance. The ChatGPT app then adds more moving parts: it may search the web, it may use what it remembers about you, and the model that answers can differ by plan and by day. This post explains each cause in plain terms, with OpenAI’s own documentation, and then what it means if you are trying to measure whether ChatGPT recommends your brand.

The short answer: ChatGPT samples its words

A language model writes in tokens: words or parts of words. At each step it scores every possible next token by how likely it is to fit, then picks one. If it always took the single most likely token, it would be predictable and often dull. So it samples: likely tokens are picked most of the time, less likely ones some of the time. Once one early word differs, everything after it is written from a different starting point, and the two answers drift apart.

A useful way to picture it: for some questions, one continuation is so much more likely than any other that the model takes it nearly every time. For others, several continuations are close, and the model wanders between them. That is why a factual question such as a date often comes back the same, while “what is the best CRM for a small agency?” rarely does. Thinking Machines Lab gave a neat example of the first kind: across 1,000 runs, every completion began “Feynman was born on May 11, 1918, in”, and then “992 of the completions go on to generate ‘Queens, New York’ whereas 8 of the completions generate ‘New York City’”^[8]. Nearly identical, and still not identical.

Is ChatGPT deterministic?

No. In the ChatGPT app there is no temperature setting at all; OpenAI chooses the settings. Some guides tell readers to “set the temperature to 0” in ChatGPT, but that control exists only for developers using the API. And even there, two facts get in the way.

Temperature 0 is not enough. In theory, temperature 0 always takes the most likely token, so the output should repeat. Thinking Machines Lab, a research lab, showed that in practice it does not: “even when we adjust the temperature down to 0 ... LLM APIs are still not deterministic in practice”. The cause is not a random number. It is the server. Requests are processed in batches with other people’s requests, and the batch size changes the order of tiny calculations, which can change which token comes out on top. In their words, “the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies!”^[8]

Their experiment ran one prompt 1,000 times at temperature 0 on an open model and got 80 different completions; with a fix they built for the batching problem, “all of our 1000 completions are identical”^[8]. That was their own server, not ChatGPT, but it explains why a setting of 0 does not mean the same text twice. An academic study of five models set up to be deterministic, run 10 times on eight tasks, found “accuracy variations up to 15% across naturally occurring runs”, and that “none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings”^[9].

The tools for repeatability are weak. OpenAI’s seed parameter is now marked deprecated. Its reference says the system “will make a best effort to sample deterministically” and that “determinism is not guaranteed”^[2]. When it launched, OpenAI’s own cookbook promised “(mostly) consistent outputs”^[4]. And for current reasoning models, the guide tells developers that when reasoning effort is not none, they should “remove temperature, top_p, and top_logprobs”^[3]. So for the newest models, the dial most guides talk about is not there.

Five more reasons the answer changes in the ChatGPT app

Sampling explains why two runs of the same model differ. The app adds causes that sit around the model, and these often matter more for questions about products and brands.

1. It may search the web, and the results move

OpenAI says “ChatGPT may search the web automatically when your question would benefit from current information”, and that it “typically rewrites your query into one or more targeted queries” that it sends to its search providers^[5]. One run may search and another may not. Two searches can be worded differently and return different pages. And the pages themselves change. OpenAI also notes that “search results and citations can be incomplete, outdated, or incorrect”^[5]. Our post on where ChatGPT gets its information covers the training and search sides in detail.

2. Your location

When it searches, “ChatGPT may use an approximate location based on your IP address to provide relevant local results”. A VPN changes that, and “if memory is enabled, saved location information may also influence search results”^[5]. Two people in different cities can get different recommendations for the same question.

3. Memory and chat history

With memory on, “ChatGPT can remember relevant preferences and details from your chats and other available sources”, and “decides which available information is relevant to a response”. With chat history referencing on, it can “use relevant information from past conversations to personalize future responses”^[6]. If you once told ChatGPT you run a small agency on a tight budget, that can shape the next answer about software, without you repeating it.

4. Which model answers

ChatGPT is not one model. Plans have different defaults, and OpenAI’s model page notes that “if you reach a Thinking limit, ChatGPT may continue with another available model”, and that “ChatGPT can still switch automatically for safety”^[7]. The lineup itself changes: the same page announces new defaults as they roll out. A question asked on a free plan and on a paid one, or this month and next, may be answered by different models.

5. The words and the conversation around them

“Best CRM for agencies” and “which CRM should a 10-person marketing agency use to track client deals?” are the same need to a person and different prompts to a model. Earlier messages in the same chat count too: everything above your question is part of what the model reads before it answers.

Does ChatGPT give the same answer to everyone?

No, and it is worth separating what changes between two runs for one person from what changes between two people:

So “ChatGPT recommends X” is never quite a fact about ChatGPT. It is a fact about one answer, to one wording, for one person, on one model, on one day.

How much the answers move: what the studies found

For general questions, the variation is mostly in wording. For questions that end in a list of brands, products or sources, it is large. Four studies, of different kinds:

• Brand lists (SparkToro and Gumshoe.ai). “600 volunteers ran 12 different prompts through each of the 3 tools a combined 2,961 times” in November and December 2025. There was “a <1 in 100 chance” that ChatGPT or Google’s AI would give the same list of brands in any two responses, and for the same order “it’s more like 1 in 1,000 runs”. List length moved too: “sometimes the AI gave as few as 2-3 recommendations, and equally often 10 or more”^[10]. Gumshoe.ai sells AI tracking, which the article discloses.
• Cited sources, week to week (SISTRIX, vendor). Across 82,619 prompts over 17 weeks, “Google replaces 56% of sources in AI-generated responses every week, whilst ChatGPT replaces as much as 74%”. For ChatGPT search, “the median prompt does not contain a single domain that appears in all 17 weeks”^[11].
• Cited sources, same day (SE Ranking, vendor). 10,000 keywords run through Google AI Mode three times on one day: “Only 9.2% of URLs matched across three tests”^[12].
• Brands versus sources (OtterlyAI, vendor). 520 prompts run daily on seven engines for 40 to 91 days. ChatGPT and Google AI Mode “reused only about 26% of the same sources”, while brand mentions were steadier: “only 12% to 19% of brands appeared on a single day during the 30-day window and were never mentioned again”, against up to 67% for cited sources^[13].

Put together: the exact list is close to random, the sources churn fast, and yet some brands come back again and again. That last part is what makes measurement possible. The post on how ChatGPT picks products covers the SparkToro study in more depth.

What it means if you are measuring your brand

If you check ChatGPT for your brand once, you have one draw from a distribution. It might be a lucky draw or an unlucky one; you cannot tell which. Three rules follow.

Count rates, not positions. Rand Fishkin, who started out sceptical, concluded that “visibility % across dozens to hundreds of prompts run multiple times is a reasonable metric”, while “any tool that gives a ‘ranking position in AI’ is full of baloney”^[10]. The share of answers that name you is stable enough to track. Your place in any one list is not.

Use more prompts, not just more runs. OtterlyAI resampled its own data and found that “with only 10 prompts, 90% of the Brand Coverage results fell between 39% and 71%”; at 100 prompts, the range narrowed to between 51% and 60%^[13]. A handful of questions gives a number that swings by tens of points for no reason. Our guide to choosing prompts covers how to build a set, and the AI share of voice guide shows how to put a margin of error on the result.

Separate the held positions from the rotating ones. A brand named in nine of ten weekly runs for a question holds that answer. One named in three of ten is in rotation. CoreCited calls these Core and Carousel, and classifies nothing until it has 4 weekly runs, because fewer than that is not enough to tell them apart.

How to get more consistent answers from ChatGPT

You cannot make the app deterministic, but you can take out much of the avoidable variation:

• Say who you are and what you need. “For a 10-person agency, under $50 a user, must work with Gmail” leaves the model far fewer close options than “best CRM”.
• Ask for the criteria first. Asking it to list how it would judge the options, then apply them, makes the reasoning visible, and you can hold it to the same criteria next time.
• Fix the format. “A table of five options with price, best for and one drawback” varies less than open prose.
• Control personalisation. Turn memory off, or use a temporary chat; OpenAI’s memory page describes an “Unpersonalized” option for chats where you do not want your memories used^[6].
• Ask it to search, or not to. If you want current information, say so. If you want the model’s general knowledge, ask it not to browse.
• For developers: use a low temperature where the model accepts one, pin a model version, store the outputs you rely on, and test across several runs rather than one.

Questions people ask

Sources