Skip to main content
Version: devel

Community Destinations

We've got several destinations contributed by our community. They are not part of the dlt library, instead we list them below.

Caveat Emptor

Use at your own risk: These are community-maintained forks of dlt that are not officially supported by the dlt team. While they extend dlt's functionality to additional databases, they may not receive the same level of testing, maintenance, or security updates as the core library. Always review the code and test thoroughly before using in production environments.

Starrocks​

by: phaethon

Implementation of Starrocks support as a separate destination. Starrocks is an MPP (Massively Parallel Processing) database designed for real-time analytics. This adapter implements two loading methods:

  • Stream Load: Direct streaming of data into Starrocks
  • INSERT INTO SELECT FROM FILES: Loading data from S3-compatible staging storage for improved performance with large datasets

Code: https://github.com/phaethon/dlt/tree/starrocks

PR with discussion: https://github.com/dlt-hub/dlt/pull/2518

Clickhouse Distributed​

by: zstipanicev

When Clickhouse is deployed with replicas and distributed tables, standard DDL & DML statements need to be modified to work across the cluster. This implementation adds:

  • ON CLUSTER clauses to DDL statements to execute across all nodes
  • Creation of base table and distributed table pairs (base tables use ReplicatedMergeTree, distributed tables use Distributed engine)
  • Modified ALTER, DROP, DELETE, UPDATE, and TRUNCATE operations to work with both table types
  • Configuration options for cluster name, database prefixes, and table name patterns

This allows dlt to work seamlessly with sharded and replicated Clickhouse setups.

Code: https://github.com/zstipanicev/dlt/tree/feat/2200-add-clickhouse-distributed-support

PR with discussion: https://github.com/dlt-hub/dlt/pull/2573

CrateDB​

by: amotl

CrateDB is a distributed SQL database built on top of Lucene, designed for real-time analytics on large datasets. This destination adapter wraps the PostgreSQL adapter with CrateDB-specific adjustments:

  • Compatibility with CrateDB's PostgreSQL wire protocol
  • Workarounds for system column naming restrictions (underscore-prefixed columns)
  • Support for CrateDB's REFRESH TABLE statements for consistency
  • Adjusted transaction handling for CrateDB's limited transaction support

Note: This PR is in draft status and under active development.

Code: https://github.com/crate-workbench/dlt/tree/cratedb

PR with discussion: https://github.com/dlt-hub/dlt/pull/2733

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.