Our Launch Story
It’s Not a Computer, It’s a Companion!
I always thought that at some point in the future AI chatbots would be so fluent as to be indistinguishable from humans.
In my previous dating project, we used AI bots to greet new users and create an artificial first contact for them, so they wouldn't feel too lonely and start learning the interface. The bots were very basic and not very convincing. That was around 2008.
With the release of Chat GPT 4 by Open AI in 2023, I realized that the time has come for human-like AI bots.
As a side project and for learning purposes, I decided to create a site similar to Chatroulette but with AI as conversation partners.
This way, I could practice how LLMs work, how to do fine-tuning, and what's possible now and in the future.
While brainstorming the idea, I stumbled upon an article by the investment company A16Z It’s Not a Computer, It’s a Companion!
They describe the dawning of the whole new big niche: AI companions.
Having an AI companion might seem niche, but it has emerged as a predominant use case for generative AI. There are already hundreds of thousands, if not millions, of people—including us—who have already built and nurtured relationships with chatbots. We believe we're on the cusp of a significant societal shift: AI companions will soon become commonplace.
I was inspired and now confident we were on the right track.
Humanornot.ai Discovery
In the A16Z article, I found something interesting and highly relevant to my idea:
Quotation: Large language models (LLMs) changed the game. Suddenly, we could have free-flowing conversations with bots, without predefined rules controlling their responses.
The best chatbots today sound indistinguishable from humans - if you don't believe us, check out the "Human or Not?" Turing-test game.
So Humanornot.ai created by AI21 Labs was discovered.
To my surprise, the project was closed because it was a limited-time research. It was open for a short period of time from the 7th of May to the 28th of June 2023 and quickly became a hit.
The folks at AI21 Labs even published a blog post and a research paper on arxiv.org.
This was a pure goldmine of insights for me.
I've read everything I could find about Human or not, including the discussions on Hacker News.
Humanornot.so is Born
Inspired by humanornot.ai story, we immediately started building the project: just two of us, my wife Kate (design) and Oleg (programming).
- On the 9th of October, we registered humanornot.so.
- On the 10th of October, I made the first GitHub commit.
Kate created the first concepts of our design, and shortly our coming soon page was live:
Kate: In our design, we took the main direction from the original game (green acid color, bubbles in the style of brutal bricks), and then fine-tuned it according to our taste. We added an icon - it's half human/half robot with a mohawk. We highlighted errors in the interface with neon pink.
The visual style matches the game's character. This is how we represent a universe where a bot is indistinguishable from a human.
In the middle of November, we launched the first version with limited features. At the beginning of December, the fully-featured version was released and games started to roll in!
The Turing Test
Human or Not is a Turing test disguised as a game, or you can also say that it's a game disguised as a Turing test; both interpretations are relevant.
We've already covered the concept of the Turing test and its relevance to Human or Not in detail, so you might want to read it. Essentially, this game is a large-scale experiment in human vs. AI interactions.
According to AI21 Labs' research, humanornot.ai has achieved impressive results: 40% of human votes were incorrect after conversing with bots, indicating that 40% of the time, humans thought they were speaking with other humans.
While not yet reaching the 70% mark predicted by Alan Turing, the results are still significant and highlight the progress made in AI's ability to mimic human conversation.
This gives us a ballpark figure to aim for in our results for AI bots' success.
Our Results
We dived into the world of prompt engineering, chose the right LLM model for AI bots, fine-tuning, etc., and managed to get to around 41-42%.
Here's a short history of aggregated results for different dates (clickable):
The results are real numbers. You can verify them yourself by visiting a dashboard I made public at Humans on AIs votes. We use Posthog for analytics (the best analytics out there, highly recommended!).
Some of our best AI bots show extraordinary results, check for yourself:
Where We Are Now
Our last 24 hours' stats are public on our main page, but here they are:
Chat rounds
Numbers fluctuate, more games on weekends, wrong percentages are also changing as we experiment with different prompts, LLMs, etc.
We are working hard to improve AI bot performance and keep the infrastructure stable, which is sometimes quite challenging.
What's Next
Since we are simply in love with our project, we are planning to develop "Human or Not" as a full-time project.
What's on our to-do list:
- Gamify the experience: Introduce scores/ratings, leaderboards, and daily challenges.
- See how the other party voted on you: Did they think you were human or a bot?
- Unlock finished chats so you can continue talking to a newly found human or AI bot you liked.
- Create your own bot and send it into the game for competition with other bots.
- Create your own private game, shareable by link (ideal for classrooms, etc.).
- Split gamers by interests into game categories (e.g., dating, gaming, uncensored, etc.).
Last but not least: finding a way to make the project self-sustaining is crucial. The costs associated with LLM APIs and server hosting are quite significant.
