Communities

LLM evaluation is incredibly difficult, and is even more difficult in specialized domains.

In order to get the best Comments and analysis of your Threads, you can invite and build a Community of experts and practioners who can evaluate your Threads. Think of it like "open source evaluations."

Why Communities are Critical to LLM Success

Communities are starting to apply LLMs to every domain, subject, and problem. And they are even starting to make specially fine-tuned LLMs.

But to be useful and continue to grow in application and performance, these LLMs must be rigorously and exhaustively evaluated by the community experts. Armed with this knowledge, the community can continue to improve LLM performance, whether that means finding the best off-the-shelf LLM to apply to a problem or continuing to fine-tune a custom LLM. Critically, this evaluation process cannot be limited just to technical engineers, but must include the non-technical domain experts who will drive their progress forward.

Threads enable communities to come together to evaluate LLMs in ways that matter to you and your specialized domain. Threads allow domain experts to build specially designed sets of evaluation datasets to test their specialized LLMs, discuss LLM performance on them, and continually iterate.

Why does this matter? Because this process of careful and deliberate evaluation is what will drive massive improvements in LLM performance for your domain. You can’t choose, fix, or customize LLMs for your domain until you have a thorough understanding of its strengths and weaknesses within your domain. As LLMs continue to be fine-tuned for each specific domain or problem, Threads allow the community to find problems and where their LLMs are not working well, and see if new fine-tuned LLMs fix these problems.

Building Communities

The easiest way to start buildling communities is to start building Threads and inviting others to join and begin adding, comments, and expanding on Threads. Here are a few ways to build communities

Public Comments

By default, all Threads allow PeerAI users to add Comments. By creating a high quality and interesting Thread, your Thread will be promoted to the front page of the site. As users browse Threads, they will discover and begin to comment on your Threads, adding an organic source of analysis to your Threads.

Invite Colleagues and Peers

Once you create a Thread, invite your colleagues and peers directly to add, analyze, and comment on your Threads. You can control who can edit and administer your Threads, while still leaving the Threads full open to the public to add Comments. Click here for a guide to inviting others to edit and comment on your Threads. The more viewpoints and opinions you get, the better analysis you will be able to drive.

Add Additional Thread Administrators and Editors

While Threads are open to the public by default to encourage maximum Comments, you can control who has the permissions to Edit and Administer your Threads. By adding additional Editors and Administrators, you can enable these extra participants to invite their own personal connections, creating a greater source of expertise to help you analyze your Threads.

Crowd Sourcing

Active communities on Reddit, Hugging Face, and other platforms are great places to solicit Comments and input. Post a link to your Thread on one of these platforms and get the entire web involved in helping you analyze LLM responses.