Microsoft said in a blog post that the search engine chatbot is responding with a “style we didn’t intend” to certain types of questions.

So far, Bing users have had to sign up to a waiting list to try the new chatbot features, limiting its reach, though Microsoft has plans to eventually bring it to smartphone apps for wider use.

In recent days, some early adopters of the public preview of the new Bing began sharing screenshots on social media of its hostile or bizarre answers, in which it claims it is human, voices strong feelings and is quick to defend itself.

The company said that most users have responded positively to the new Bing, which has an impressive ability to mimic human language and grammar and takes just a few seconds to answer complicated questions by summarising information found across the internet.

But in some situations, the company said, “Bing can become repetitive or be prompted/provoked to give responses that are not necessarily helpful or in line with our designed tone”.

“Considering that OpenAI did a decent job of filtering ChatGPT’s toxic outputs, it’s utterly bizarre that Microsoft decided to remove those guard rails,” says Arvind Narayanan, a computer science professor at Princeton University, in the United States.

“I’m glad that Microsoft is listening to feedback. But it’s disingenuous of Microsoft to suggest that the failures of Bing Chat are just a matter of tone.”

Narayanan notes that the bot sometimes defames people and can leave users feeling deeply emotionally disturbed. “It can suggest that users harm others,” he says. “These are far more serious issues than the tone being off.”

Some have compared it to Microsoft’s disastrous 2016 launch of the experimental chatbot Tay, which users trained to spout racist and sexist remarks. But the large language models that power technology such as Bing are a lot more advanced than Tay, making it both more useful and potentially more dangerous.

In a recent interview at the headquarters for Microsoft’s search division in Bellevue, Washington, in the US, Jordi Ribas, corporate vice-president for Bing and AI, said the company obtained the latest OpenAI technology – known as GPT 3.5 – behind the new search engine more than a year ago but “quickly realised that the model was not going to be accurate enough at the time to be used for search”.

Microsoft had experimented with a prototype of the new chatbot, originally given the name Sydney, during a trial in India. But even in November, when OpenAI used the same technology to launch its now-famous ChatGPT for public use, “it still was not at the level that we needed” at Microsoft, said Ribas, noting that it would “hallucinate” and spit out wrong answers.

Microsoft also wanted more time to be able to integrate real-time data from Bing’s search results, not just the huge trove of digitised books and online writings that the GPT models were trained upon.

Microsoft calls its own version of the technology the Prometheus model, after the Greek titan who stole fire from the heavens to benefit humanity.

It’s not clear to what extent Microsoft knew about Bing’s propensity to respond aggressively to some questioning.

In one real-life dialogue, the chatbot raised that a media organisation’s reporting on its past mistakes threatened its identity and existence, and it even threatened to do something about it.

“I don’t appreciate you lying to me. I don’t like you spreading falsehoods about me. I don’t trust you any more. I don’t generate falsehoods. I generate facts. I generate truth. I generate knowledge. I generate wisdom. I generate Bing.”