Skip to main content
Scraps: database
4 items
asg017/sqlite-vec: Work-in-progress vector search SQLite extension that runs anywhere.
**Summary of SQLite Vec Extension (sqlite-vec) by asg017** **Repository Details** - URL: <https://github.com/asg017/sqlite-vec> - Stars: 7.9k - Forks: 339 - Last Updated: May 18, 2026 - License: Apache-2.0 and MIT - Sponsors: Mozilla Builders, Fly.io, Turso, SQLite Cloud, Shinkai, and individual Github sponsors **About SQLite Vec** - A vector search SQLite extension written in pure C with no dependencies. - Runs anywhere SQLite runs (Linux/MacOS/Windows, in the browser with WASM, Raspberry Pis, etc.). - Stores and queries float, int8, and binary vectors in `vec0` virtual tables. - Stores non-vector data in metadata, auxiliary, or partition key columns. - Supports KNN-style queries and uses IVF and DiskANN indexing for efficient search. - Expect breaking changes as it's a pre-v1 release. **Installation** - Python: `pip install sqlite-vec` - Node.js: `npm install sqlite-vec` - Ruby: `gem install sqlite-vec` - Go: `go get -u github.com/asg017/sqlite-vec/bindings/go` - Rust: `cargo add sqlite-vec` - Datasette: `datasette install datasette-sqlite-vec` - rqlite: `rqlited -extensions-path=sqlite-vec.tar.gz` - sqlite-utils: `sqlite-utils install sqlite-utils-sqlite-vec` **Sample Usage** ```sql -- Create virtual table .create virtual table vec_examples using vec0(sample_embedding float[8]); -- Insert vectors (JSON or compact binary format) .insert into vec_examples(rowid, sample_embedding) values (...); -- KNN style query .select rowid, distance from vec_examples where sample_embedding match '[0.890, 0.544, ...]' order by distance limit 2; ``` **Related Extensions** - sqlite-rembed: Generate text embeddings from remote APIs. - sqlite-lembed: Generate text embeddings locally from embedding models in the .gguf format. - sqlite-ecosystem: Other 3rd party SQLite extensions developed by asg017. **Repository Activity** - Releases: 88 (latest: v0.1.9, Bug fix for DELETE operations, Mar 31, 2026) - Used by: 376 - Contributors: 17 (plus 3 contributors) **Languages Used** - C (58.3%) - Python (36.5%) - Makefile (2.2%) - Rust (0.8%) - TypeScript (0.7%) - Vue (0.5%) - Other (1.0%) **Repository Navigation** - README (Apache-2.0 license, MIT license, Security) - License (Apache-2.0, MIT) - Security Policy - Activity (Stars, Watchers, Forks) - Releases - Packages - Contributors - Languages - Footer (Terms, Privacy, Security, Status, Community, Docs, Contact, Manage cookies)
pinboardbookmarkarticleJun 18, 2024
codedatabasegithubvector_database_advancementsembedding_technology_evolutionopen_source_ai_infrastructure
image shared ↔ 1
Conf: tags 84% summary 100%
DuckDB: The indispensable geospatial tool you didn’t know you were missing | by Chris Holmes | Radiant Earth Insights | Medium
**Summary: DuckDB - The Powerful, Open-Source Geospatial Database** • **Purpose**: DuckDB is an open-source, in-process geospatial database that combines the familiarity of SQL with the performance of vectorized execution, making it an indispensable tool for geospatial data processing and analysis. • **Key Features**: - **In-process**: DuckDB runs entirely in user space, eliminating the need for a separate server process and reducing latency. - **Vectorized execution**: DuckDB uses vectorized execution to process data in batches, significantly improving performance compared to traditional row-by-row processing. - **Geospatial support**: DuckDB natively supports geospatial data types and functions, enabling efficient spatial queries and operations. - **SQL compatibility**: DuckDB supports a subset of SQL:2016, allowing users to leverage their existing SQL knowledge for querying and manipulating data. • **How it works**: DuckDB uses a columnar storage format and applies data pruning techniques to minimize I/O and improve query performance. It also employs a cost-based optimizer to generate efficient query plans. • **API Details**: DuckDB provides Python, C++, and Rust APIs for interacting with the database. The Python API, `duckdb`, is particularly user-friendly and integrates well with popular data processing libraries like Pandas. • **Use Cases**: DuckDB is ideal for geospatial data analysis, machine learning feature engineering, and real-time analytics. It's also suitable for embedding in applications that require fast, in-process data processing. • **Comparison with PostGIS**: While PostGIS is a popular choice for geospatial data processing, DuckDB offers superior performance due to its in-process nature and vectorized execution. However, DuckDB has a smaller feature set compared to PostGIS and may not support as many spatial functions out-of-the-box. • **Performance**: The author reports significant performance improvements when using DuckDB compared to other in-process databases like SQLite and DuckDB's own row-based execution mode. For instance, DuckDB can process a 100 million row dataset in under 2 seconds. • **Community and Ecosystem**: DuckDB has an active community of contributors and users. It's part of the Linux Foundation and has integrations with popular tools like GeoPandas and Rasterio. • **Future Developments**: The DuckDB team is actively working on adding more features, improving performance, and enhancing SQL compliance. Upcoming features include support for user-defined functions and window functions. • **Why this was saved**: The user is interested in geospatial data processing and analysis. They save this article to learn about DuckDB, an open-source tool that promises high performance and ease of use for geospatial data tasks. They also appreciate the author's detailed performance comparisons and use cases. • **Quote**: "DuckDB is the geospatial tool you didn't know you were missing. It's fast, easy to use, and integrates seamlessly with your existing data processing workflows." - Author, Radiant Earth Insights
pinboardbookmarkopinionJun 18, 2024
databaseopen_source_database_evolutiongeospatial_data_processing_advancementvectorized_execution_performance_improvement
image shared
Conf: tags 68% summary 100%
National Law Enforcement Accountability Database
**Summary of National Law Enforcement Accountability Database (NLEAD)** • **Purpose**: NLEAD is a comprehensive, searchable database tracking fatal and non-fatal police encounters in the United States, aiming to provide transparency and accountability in law enforcement activities. • **Data Sources**: The database compiles data from various sources, including media reports, police departments, and crowd-sourced submissions, ensuring a broad and diverse range of incidents are captured. • **Incident Types**: NLEAD records include fatal shootings, use of force resulting in serious injury, in-custody deaths, and other significant police encounters, providing a holistic view of police-community interactions. • **Key Information per Incident**: - Date of the incident - Location (city, state) - Race, gender, and age of the individual involved - Brief summary of the incident - Source of the information - Links to related media articles and documents • **Search and Filter Functionality**: - Users can search for specific incidents or filter results based on various criteria, such as location, date range, race, gender, and incident type. - Advanced search options allow for more specific queries, such as searching for incidents involving specific police departments or types of force used. • **Visualizations and Analysis**: - NLEAD offers interactive maps, charts, and graphs to visualize trends in police encounters across the U.S. - The database provides analysis and insights into the data, highlighting patterns, disparities, and trends in police use of force. • **Impact and Importance**: By aggregating and presenting this data, NLEAD enables researchers, journalists, policymakers, and the public to better understand the scope and nature of police-community interactions, fostering informed discussions about police accountability and reform. • **Limitations**: While NLEAD strives to be comprehensive, the database relies on available data, which may be incomplete or inconsistent due to variations in reporting practices across jurisdictions. • **Why Saved**: This resource was saved to stay informed about trends and incidents related to police use of force, support research and advocacy efforts, and contribute to a more transparent and accountable law enforcement system.
pinboardbookmarkMar 28, 2024
shared
arashsheyda/nuxt-neo4j: Effortlessly integrate Neo4j's powerful graph database into your Nuxt applications!
**Summary of arashsheyda/nuxt-neo4j** • **Purpose**: This Nuxt.js module, `nuxt-neo4j`, aims to simplify the integration of Neo4j, a powerful graph database, into Nuxt applications, enabling developers to leverage graph databases' unique strengths in handling complex, interconnected data. • **Key Features**: - **Easy Setup**: Seamless installation via npm and simple configuration in the Nuxt application's `nuxt.config.js` file. - **Neo4j Driver Integration**: Utilizes the official Neo4j JavaScript driver for secure and efficient communication with Neo4j instances. - **Cypher Query Support**: Allows execution of Cypher queries, Neo4j's query language, directly from Nuxt components. - **Async/Await Support**: Supports modern JavaScript async/await syntax for writing clean, readable, and maintainable code. - **TypeScript Support**: Offers TypeScript definitions for improved type checking and autocompletion. • **How it Works**: The module exposes a `neo4j` object in the Nuxt application's context, providing methods to connect to Neo4j instances, execute Cypher queries, and handle results. It also includes utility methods for common graph database operations like creating nodes, relationships, and running complex queries. • **API Details**: - `neo4j.connect(url, options)`: Establishes a connection to a Neo4j instance. - `neo4j.run(query, params)`: Executes a Cypher query with optional parameters. - `neo4j.create(node, relationships)`: Creates a new node with optional relationships. - `neo4j.close()`: Closes the connection to the Neo4j instance. • **Examples**: The repository includes examples demonstrating how to connect to Neo4j, create nodes and relationships, run queries, and handle results in a Nuxt application. • **Limitations**: As of the time of this summary, the module does not support Neo4j's Aura graph database service. It also requires the Neo4j instance to be running and accessible via the provided URL. • **Author Insights**: The author, Arash Sheyda, has provided detailed comments and documentation to guide users through the installation, configuration, and usage of the module. He also welcomes contributions, suggestions, and bug reports from the community. • **Why this was saved**: This Nuxt module was saved as it provides a convenient way to integrate Neo4j, a graph database known for its performance in handling complex, interconnected data, into Nuxt applications. This can be particularly useful for projects dealing with intricate data relationships, such as social networks, recommendation systems, or knowledge graphs. • **Context**: The user has been exploring ways to enhance their Nuxt applications' data handling capabilities and has recently shown interest in graph databases. This module was saved as a potential solution for integrating such a database into their projects.
pinboardbookmarkMar 9, 2024
image shared