Let's talk

Choosing your model

Open Source vs Closed Source LLM

Deciding between open-weight and closed models depends on your requirements for data residency, cost, and ease of use.

By Divyaprakash JhaUpdated 5 min readTopic: Large language models

Photo for Open Source vs Closed Source LLM

The Core Difference

Closed models are available only through a provider's API or specific apps, making them easy to start with but requiring you to send data to an external provider. Open-weight models allow you to download and run the model on your own servers or devices, offering greater control over data privacy and customization.

When choosing between these two, the decision often comes down to whether you want the convenience of a managed service or the sovereignty of hosting your own infrastructure.

  • Closed models: You access them via an API. You do not own the model, and you must follow the provider's pricing and terms.
  • Open-weight models: The trained weights are published, allowing you to run them locally. This is ideal for privacy-sensitive tasks.

Closed Source Models: Ease and Capability

Closed models are typically the most capable and easiest to start with for developers. If you want to add AI to your business tomorrow, a closed model via API is the fastest route. You don't need to worry about buying expensive GPUs or managing complex software stacks; you simply send a request and get a response.

However, this convenience comes with trade-offs:

  • Dependency: You depend on the provider's pricing structures. If they raise their prices or change how the model behaves, your application may be affected.
  • Data Privacy: Your data is sent to the provider to process the request. For businesses with strict compliance needs, this is a critical factor.
  • Black Box: You cannot see exactly how the model was trained or fine-tune its core weights to fit your specific niche.

For many startups, starting with a closed model is the best way to prove a concept before committing to the overhead of managing their own hardware.

Open-Weight Models: Control and Privacy

Open-weight models are the preferred choice for organizations that require high levels of control. Because you host the model, you have total authority over where your data goes.

These models are ideal for several specific business use cases:

  • Data Residency: If your industry has strict rules stating that data cannot leave your servers or your specific country, open-weight models are often the only viable path.
  • Offline Use: They are best for environments with patchy connectivity or for applications that must work entirely offline.
  • High-Volume Work: If you are processing millions of requests, you might find that hosting your own model is more cost-effective than paying a per-query fee to an API provider.
  • Deep Customization: You can perform fine-tuning on your own hardware to teach a model a very specific style or a narrow set of skills.

If you are interested in how these models actually function, you can read more in our guide on how LLMs work.

How to Choose the Right Model

Choosing a model is not just about picking the one that sounds smartest. A practical approach involves testing the model against your actual business needs.

1. Define your task with a test set

Do not rely on public leaderboards alone. Collect a test set of 30 to 50 real examples of your data. These should include the input you expect to see and the "gold standard" answer you want the model to produce. Run your candidate models against this set to see which one actually performs for your specific task.

2. Check language support

If your customers communicate in regional languages like Hindi, Marathi, or Hinglish, you must test specifically for them. Language quality varies significantly between models, and a model that excels at English might struggle with the nuances of Indian regional scripts.

3. Start large, then shrink

A common mistake is trying to use the smallest, cheapest model from day one. Instead, start with a large, highly capable model to get your workflow and prompts working correctly. Once you know exactly what the task requires, test if a smaller or quantized model can handle the job. This "top-down" approach saves significant engineering time.

4. Weigh total cost and latency

Don't just look at the cost per token. Look at the cost per task. A reasoning model that solves a complex problem in one try might be much cheaper and faster than a cheaper, faster model that requires three different prompts to get the job done correctly.

Quantization and Hardware

If you decide to host your own models, you will quickly realize that "size" matters. Large models require massive amounts of memory (VRAM). This is where a technique called quantization becomes essential for builders.

Quantization reduces the memory requirements of a model by storing its weights in fewer bits. For example:

  • 8-bit quantization: Roughly halves the memory required compared to standard 16-bit precision, with very little loss in quality.
  • 4-bit quantization: Roughly quarters the memory requirement. This is often the "sweet spot" for running powerful models on consumer-grade hardware.

Using formats like GGUF allows you to run these quantized models efficiently on a variety of hardware, including CPUs, Apple silicon, and GPUs. This technology even allows models to run fully offline on recent mobile phones. This is incredibly useful for users in areas with patchy connectivity or for applications where data privacy is the highest priority.

To understand the underlying math of how these models use hardware, see our guide on LLM implementation.

Models for AI Agents

The next wave of AI involves "agents"—models that don't just talk, but actually *do* things. An agent uses tool use (also known as function calling) to interact with the real world. For example, instead of just saying "I will check your stock," an agent can actually call a function like check_stock(sku_id) to get real-time data from your database.

If your roadmap includes building agents, your selection criteria must change. You should prioritize models that demonstrate strong capabilities in:

  • Tool Calling: The ability to correctly format a request to an external function.
  • Structured Output: The ability to respond in a strict format like JSON, which your code can easily parse.

An agent might help a small business owner reconcile payments or follow up on unpaid invoices over WhatsApp. For these multi-step, high-stakes tasks, you need a model that can plan, execute, and react to the results of its actions.

Frequently Asked Questions

Is there a difference between open source and open-weight LLMs?
Yes. "Open-weight" is more accurate because while you can download and run the model, the original training data and code are often not released.
Can I run an LLM offline?
Yes, open-weight models can be run on your own servers or even on mobile phones using quantization to reduce memory needs.
Which is better for data privacy?
Open-weight models are generally better for privacy because you can host them on your own infrastructure, ensuring data does not leave your control.
Why are closed models often easier to use?
Closed models are provided via ready-to-use APIs, removing the need for you to manage the underlying hardware or complex deployment.
Does model size affect the choice?
Yes. You can start with large, powerful models to prove a concept, then move to smaller, cheaper, or quantized models once the task is defined.
Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

Is there a difference between open source and open-weight LLMs?

Yes. 'Open-weight' is more accurate because while you can download and run the model, the original training data and code are often not released.

Can I run an LLM offline?

Yes, open-weight models can be run on your own servers or even on mobile phones using quantization to reduce memory needs.

Which is better for data privacy?

Open-weight models are generally better for privacy because you can host them on your own infrastructure, ensuring data does not leave your control.

Why are closed models often easier to use?

Closed models are provided via ready-to-use APIs, removing the need for you to manage the underlying hardware or complex deployment.

Does model size affect the choice?

Yes. You can start with large, powerful models to prove a concept, then move to smaller, cheaper, or quantized models once the task is defined.