As a developer, we often get confused about which tools to use to build our next (side) project. Even if you're serious enough to build a product, the problem remains the same. So, I'm covering 9 open-source libraries that will truly supercharge your future project!
1. Crawlee - Web Scraping, Simplified and Powerful
Crawlee is a complete web scraping and browser automation library that helps you build reliable crawlers quickly and efficiently.
With its built-in anti-blocking features, you can create human-like bots which reduce the chances of getting blocked. Whether you're using Node.js or Python, Crawlee gives a unified interface for HTTP and headless browser crawling.
The easiest way to start using Crawlee is with the Crawlee CLI:
npx crawlee create my-crawler
This sets up a new project with all the dependencies you need. If you want to add Crawlee to an existing project:
npm install crawlee playwright
Here's a quick example of what you can do:
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, enqueueLinks, pushData, log }) {
const title = await page.title();
log.info(`Title of ${request.loadedUrl} is '${title}'`);
await pushData({ title, url: request.loadedUrl });
await enqueueLinks();
},
maxRequestsPerCrawl: 20,
});
await crawler.run(['https://crawlee.dev']);
await crawler.exportData('./result.csv');
Link: https://crawlee.dev/
2. Encore - Backend Framework for Type-Safe Applications
Managing cloud services for scalable applications is never easy. It often gets messy with complex infrastructure, inconsistent APIs, and scattered DevOps processes.
Encore simplifies this chaos by providing a unified development platform integrating type-safe backend frameworks, automatic infrastructure provisioning, and DevOps automation.
Available in both Golang and TypeScript, you can create production-ready backends using primitives like microservices, Postgres, and Pub/Sub, all without the usual complexity.
Get started by installing the CLI:
curl -L https://encore.dev/install.sh | bash
Then create an app:
encore app create
Here's what a simple API looks like:
import { api } from "encore.dev/api";
export const world = api(
{ method: "GET", path: "/hello/:name", expose: true },
async ({ name }: { name: string }): Promise<Response> => {
return { message: `Hello ${name}!` };
},
);
interface Response {
message: string;
}
Encore automatically handles authentication, HTTP routing, request validation, error handling, observability, and more.
Link: https://encore.dev/
3. Taipy - Build AI Web Apps in Python
Have you ever wanted to build a simple AI project without complicating it with a front-end framework? If so, Taipy is for you.
Taipy is an open-source library for building production-ready AI apps faster in Python. It lets you go from simple pilots to production-ready web applications quickly by building components, interfaces, and dashboards using Python commands.
Getting started is easy:
pip install taipy
Here's a simple example of creating a GUI with Taipy:
from taipy.gui import Gui
import taipy.gui.builder as tgb
from math import cos, exp
value = 10
def compute_data(decay:int)->list:
return [cos(i/6) * exp(-i*decay/600) for i in range(100)]
def slider_moved(state):
state.data = compute_data(state.value)
with tgb.Page() as page:
tgb.text(value="# Taipy Getting Started", mode="md")
tgb.text(value="Value: {value}")
tgb.slider(value="{value}", on_change=slider_moved)
tgb.chart(data="{data}")
data = compute_data(value)
if __name__ == "__main__":
Gui(page=page).run(title="Dynamic chart")
Link: https://taipy.io/
4. KitOps - Manage AI Models with Flexibility
KitOps is an open-source tool designed to unify how data scientists, developers, and SREs manage and collaborate on AI/ML models, source codes, datasets, and other artifacts.
It brings flexibility to AI/ML model management, transforming the way teams package, version, and deploy their resources using OCI standards for maximum compatibility.
KitOps lets you package models, datasets, and code as OCI-compliant ModelKits for easy versioning. It's compatible with any OCI-compliant registry and uses YAML-based configuration for fine-tuning LLMs and creating RAG pipelines.
Link: https://dub.sh/kitops
5. CopilotKit - Build In-App AI Agents Easily
If you're trying to add AI to your application, CopilotKit is probably the simplest and most production-ready way to do so.
CopilotKit is the leading open-source framework for building in-app AI copilots. You can build in-app AI chatbots and AI agents with simple components.
The In-App AI Chatbot feature lets you add an AI chatbot to your app with easy plug-and-play components. With Copilot Readable State, your copilots can access application state. Copilot Actions enable your AI assistants to perform tasks directly within the application.
They provide built-in UX components like <CopilotKit />, <CopilotPopup />, <CopilotSidebar />, and <CopilotTextarea />.
Link: https://www.copilotkit.ai/
6. Tolgee - Web-based Localization Platform
Internationalization and localization are crucial for reaching a global audience, but managing both manually can be very time-consuming.
Tolgee is an open-source localization platform that makes it incredibly easy to translate your applications into any language without modifying your code. It's perfect for web, mobile, and desktop applications.
The In-Context Translation feature lets you see translations exactly as they'll appear in your app. The platform helps you manage translation keys, set up roles and permissions, and extend functionality with plugins.
Tolgee provides SDKs, a REST API, and CLI to make localization easier across different environments.
Link: https://tolgee.io/
7. OPAL - Administration Layer for Policy Engine
OPAL is an open-source administration layer designed to work with policy engines like Open Policy Agent (OPA) and AWS Cedar.
If you're working on large projects with complex access controls, OPAL simplifies the process of managing permissions across teams or user groups.
It detects changes to policies and policy data in real-time, ensuring your app's authorization layer stays up-to-date. Whether your data changes via APIs, Git, or third-party services, OPAL automatically syncs the necessary policies to your services.
OPAL's Real-Time Permission Updates automatically grant or revoke permissions in real-time. With Policy as Code, developers can write policies as code, making them easier to version and manage.
Link: https://io.permit.io/se-opal
8. ChartDB - Visualize Databases Instantly
ChartDB is a powerful, web-based tool that helps developers visualize and manage their database schemas quickly and intuitively.
The Instant Schema Import feature lets you run a single "Smart Query" to retrieve your database schema as JSON instantly.
What I love is the AI-Powered Export for Migration. ChartDB's AI can generate DDL scripts tailored to your target database dialect, making migrations between databases practically seamless.
The Interactive Schema Editor lets you easily fine-tune your schema layout, adjust structures, and make complex database structures clearer and easier to manage.
ChartDB's dashboard brings everything together, letting you control multiple database schemas in real-time and export them in SQL or image format for sharing.
Link: https://chartdb.io/
9. Stack Auth - Managed User Authentication
Stack Auth is an open-source alternative to Auth0 and Clerk. If you're looking to integrate a self-hosted, customizable user authentication system, Stack Auth is an excellent choice, as its self-hosted version is completely free.
It has extensive support for features like single sign-on (SSO), OAuth, and multi-factor authentication (MFA). It's particularly suited for projects where you need to avoid vendor lock-in or reduce costs associated with third-party auth services.
The ability to self-host ensures that sensitive user data stays within your infrastructure, offering enhanced security for applications with strict compliance needs.
Link: https://github.com/stack-auth/stack
Conclusion
These nine open-source libraries cover a wide range of development needs, from web scraping and backend frameworks to AI integration and authentication. By leveraging these tools, you can significantly reduce development time, improve code quality, and focus on building features that matter to your users. The best part? They're all free, actively maintained, and backed by strong communities.
