Skip to content

Data Operations (CRUD)

Use mo_crud_kit for model-aware bulk create/read/update workflows with Polars frames. This workflow is designed for data-heavy endpoints where serializer-by-row patterns become a bottleneck. It applies directly to high-volume ingestion and update APIs.

When to use mo_crud_kit — and when not to

mo_crud_kit is designed for workflows where data arrives or is consumed in bulk: ingestion pipelines, large exports, high-volume upserts, or any operation where the dataset is large enough that row-by-row serializer overhead or bulk_create become the bottleneck. It works on Polars DataFrame and LazyFrame inputs — not on individual model instances.

For everyday operations — creating a single record, updating a user profile, returning a list of 20 results — use native Django serializers. They are simpler, easier to debug, and the right tool at that scale. Reach for mo_crud_kit when volume is the problem. The two approaches are complementary; most projects use both.

Prerequisites

  • Models are migrated and uses UUID primary/foreign keys.

Implementation

mo_crud_kit provides model-aware operations for bulk create, read, and update. It uses model metadata plus Polars validation to keep writes fast and consistent.

1. Model Frames

mo_crud_kit works on a model-frame mapping:

{
    OrderModel: order_df_or_lazy,
    OrderItemModel: item_df_or_lazy,
}

Each write operation returns:

  • status: ok, partial_ok, or fail
  • valid_model_frms: rows that passed validation
  • invalid_model_frms: rows with error metadata

Sample Model Frame

import polars as pl
from apps.orders.models import OrderModel

order_df = pl.DataFrame(
    {
        "order_id": ["f2fa1a5b7abf4f37a3f7e14725c0b211"],
        "status": ["draft"],
        "total_amount": [120.50],
    }
)

model_frms = {
    OrderModel: order_df,
}

2. Create

Create rows from model-to-frame mappings with optional validation pipeline.

Usage:

from django_mindoff import mo_crud_kit

status, valid_model_frms, invalid_model_frms = mo_crud_kit.create(
    {
        OrderModel: order_df,
        OrderItemModel: order_item_df,
    },
    is_partial=False,
    validation_level="full",
    batch_size=1000,
)

Parameters:

  • model_frms (dict[type[models.Model], pl.DataFrame|pl.LazyFrame]): Input model-frame mapping for bulk insert.
  • is_partial (bool, default=False): If True, allows partial save when only a subset of rows are valid (applies to validation_level="full" only).
  • validation_level ("full"|"columns_only"|"none", default="full"): Selects how much validation runs before the insert.
  • batch_size (int, default=1000): Maximum number of rows written per INSERT batch.

Varieties:

  • validation_level="full" (default): runs ColumnValidator -> RowValidator -> ForeignKeyValidator and honors is_partial.
  • validation_level="columns_only": runs only ColumnValidator to normalize frame shape (rename to db_column, add missing/auto columns, drop extras) and writes directly, trusting the caller's row values and foreign keys. Row-level passes are skipped, so field defaults, auto_now/auto_now_add timestamps, UUID generation, and type coercion are NOT applied — the caller must supply database-ready values for every required column. is_partial does not apply (no per-row invalidation).
  • validation_level="none": skips all validation (including column normalization) and writes the frames as-is; emits an unsafe-write warning.
  • Partial-save mode (full only): is_partial=False fails if any invalid rows exist; is_partial=True saves valid rows and returns invalid rows separately.

Possible responses:

  • Returns ("ok", valid_model_frms, {}) when all rows are valid and inserted.
  • Returns ("partial_ok", valid_model_frms, invalid_model_frms) when partial mode is enabled and some rows are invalid.
  • Returns ("fail", valid_model_frms, invalid_model_frms) when validation fails and no write should proceed.

Notes:

  • Invalid rows contain an error column (POLARS_VALIDATOR_ERROR_COL or __error__info).
  • validation_level="none" skips safety checks and may persist unsafe data.
  • Rows are written with each backend's fastest same-transaction bulk loader — PostgreSQL COPY, MySQL LOAD DATA LOCAL INFILE (auto-falling back to row binding when the server forbids it), SQLite executemany — so the columnar frame reaches the database without a per-row Python detour, and the whole create stays atomic.

3. Read

Read queryset data into Polars with streaming/pagination variants.

Usage:

from django_mindoff import mo_crud_kit

frm, stats = mo_crud_kit.read(
    OrderModel.objects.filter(is_active=True).values(),
    page_number=1,
    is_lazy=False,
    batch_size=100,
)

Parameters:

  • qs (models.QuerySet): Queryset that must use .values() output.
  • page_number (int|None, default=None): When provided, enables pagination mode. When None, uses streaming mode.
  • is_lazy (bool, default=False): If True, returns pl.LazyFrame; otherwise returns pl.DataFrame. In streaming mode with the auto/connectorx engines this is a real larger-than-RAM scan: rows are streamed to a temporary Parquet file and the returned LazyFrame scans it lazily (the temp file is removed when the frame is garbage-collected). The iterator engine keeps the legacy deferred-.lazy() behavior.
  • batch_size (int, default=0): Chunk/page size. Auto-resolved when 0. Drives the lazy-scan/streaming fetch size and pagination; eager fast-path reads fetch in one transfer.
  • with_stats (bool, default=True): When True, computes total_count/total_pages (and the empty-set short-circuit) via extra exists()/count() queries. Set False for the fastest read: those queries are skipped, total_count/total_pages are None, and pagination derives has_next by fetching one extra row.
  • json_column_mode ("auto"|"object"|"text", default="auto"): How JSONField columns are returned. text keeps raw JSON text (Utf8); object parses to pl.Object; auto resolves to text.

Varieties:

  • Streaming mode (page_number=None): reads the full dataset.
  • Pagination mode (page_number=<n>): reads one page and returns paging metadata.
  • Materialization mode: eager (DataFrame) or lazy (LazyFrame).

Possible responses:

  • Returns (frm, stats) where frm is DataFrame/LazyFrame and stats includes: mode, batch_size, total_count, total_pages, current_page, has_next, has_previous.
  • Returns empty frame with zeroed stats for empty querysets.
  • Raises validation error if queryset is not .values()-based.

Note:

  • Frames are normalized to the canonical model dtypes (_crud_kit.dtypes.DJANGO_TO_POLARS_TYPE_MAP), the same mapping the create/update validators enforce, so a read can be fed straight back into update.

4. Read Batches

Stream a queryset as an iterator of Polars frames (memory-bounded).

Usage:

from django_mindoff import mo_crud_kit

for frm in mo_crud_kit.read_batches(
    OrderModel.objects.filter(is_active=True).values(),
    batch_size=10_000,
):
    process(frm)

Parameters:

  • qs (models.QuerySet): Queryset that must use .values() output.
  • batch_size (int, default=0): Rows per yielded frame. Auto-resolved to 1000 when 0.
  • json_column_mode ("auto"|"object"|"text", default="auto"): JSON handling, identical to read(). Each batch is streamed from Django's cursor and normalized to the canonical model dtypes.

Possible responses:

  • Returns a generator of pl.DataFrame chunks; never concatenated, so the full result set is never held in memory at once.
  • Raises validation error if queryset is not .values()-based.

5. Update

Upsert rows from model-to-frame mappings via a staged merge.

Usage:

from django_mindoff import mo_crud_kit

status, valid_model_frms, invalid_model_frms = mo_crud_kit.update(
    {
        OrderModel: order_updates_df,
    },
    is_partial=True,
    validation_level="full",
    batch_size=1000,
)

Parameters:

  • model_frms (dict[type[models.Model], pl.DataFrame|pl.LazyFrame]): Input model-frame mapping for bulk update/upsert.
  • is_partial (bool, default=False): If True, allows valid rows to proceed even when invalid rows exist (applies to validation_level="full" only).
  • validation_level ("full"|"columns_only"|"none", default="full"): Selects how much validation runs before the upsert. Missing DB columns are auto-fetched by primary key first (unless skip_db_fill=True), regardless of level.
  • batch_size (int, default=1000): Batch size used for missing-column fetch and update processing.
  • skip_db_fill (bool, default=False): When True, skips the missing-column prefetch (the per-PK SELECT that back-fills columns the frame omits). Use it when the caller already supplies every column to be written — e.g. a frame from read() that was modified in place — to avoid the extra round-trip. Primary-key canonicalization and the PK presence check still run, so upsert matching is unaffected. Caution: any column the frame omits is NOT back-filled and will be written as NULL, so only enable this when the frame is column-complete.

Varieties:

  • validation_level="full" (default): runs ColumnValidator -> RowValidator -> ForeignKeyValidator and honors is_partial.
  • validation_level="columns_only": runs only ColumnValidator (after the missing-column fetch) and upserts directly, trusting the caller's row values and foreign keys. Row-level passes are skipped, so the caller must supply database-ready values.
  • validation_level="none": skips all validation; upserts as-is and emits an unsafe-write warning.

Possible responses:

  • Returns ("ok", valid_model_frms, {}) when all rows are valid and updated.
  • Returns ("partial_ok", valid_model_frms, invalid_model_frms) when partial mode is enabled and some rows are invalid.
  • Returns ("fail", valid_model_frms, invalid_model_frms) when validation blocks update.

Notes:

  • Missing DB columns are auto-fetched using primary key before validation unless skip_db_fill=True.
  • Invalid rows include model-aware error details in error column.
  • The staging merge loads the staging table with the same fast same-transaction bulk loader as create() (PostgreSQL COPY, MySQL LOAD DATA LOCAL INFILE, SQLite executemany) and then merges set-based in SQL, so no per-row Python materialization occurs.

Example Usage

from django_mindoff import mo_crud_kit
from apps.orders.models import OrderModel

# CREATE (full validation, written in batch_size chunks via the backend's fast
# bulk loader: PostgreSQL COPY / MySQL LOAD DATA / SQLite executemany)
create_status, create_valid, create_invalid = mo_crud_kit.create(
    {OrderModel: order_df},
    validation_level="full",   # "full" | "columns_only" | "none"
    is_partial=False,
    batch_size=1000,
)

# CREATE (fast path: caller guarantees clean, DB-ready rows — skip row+FK passes)
mo_crud_kit.create(
    {OrderModel: order_df},
    validation_level="columns_only",
)

# READ (queryset must use values())
orders_frm, stats = mo_crud_kit.read(
    OrderModel.objects.filter(is_active=True).values(),
    page_number=1,
    batch_size=100,
    with_stats=False,   # skip exists()/count() for the fastest read
)

# LARGER-THAN-RAM: a real lazy scan (streamed to disk, scanned lazily)
lazy_frm, _ = mo_crud_kit.read(
    OrderModel.objects.all().values(), is_lazy=True
)

# LARGER-THAN-RAM: process in memory-bounded chunks (never concatenated)
for chunk in mo_crud_kit.read_batches(
    OrderModel.objects.all().values(), batch_size=10_000
):
    handle(chunk)

# UPDATE (staged merge: bulk-load staging table, then set-based SQL merge)
update_status, update_valid, update_invalid = mo_crud_kit.update(
    {OrderModel: order_df},
    validation_level="full",   # "full" | "columns_only" | "none"
    is_partial=True,
    # skip_db_fill=True,  # skip the missing-column prefetch when the frame
    #                     # already has every column (e.g. a full read() frame)
)

Core Concepts

1. Polars Serialization

mo_crud_kit uses model metadata plus Polars validators (ColumnValidator, RowValidator, ForeignKeyValidator) to sanitize and validate rows before DB writes. This is the intended replacement for serializer-driven bulk validation in data-heavy pipelines.

What this gives you:

  • Type normalization aligned with Django field definitions.
  • Constraint checks (required/nullability, choices, min/max, length, FK consistency).
  • Structured invalid-row capture in the configured error column (POLARS_VALIDATOR_ERROR_COL, default __error__info).

Validate before write

mo_crud_kit is built for validated tabular data. If you choose to skip the inbuilt validation + serialization, cover request-level validation in API code before sending it to CRUD Kit.

2. Limitations

  • ManyToManyField is not supported in row validation.
  • BinaryField is not supported in row validation.
  • Any Django field not mapped in row validator dtype map is unsupported.
  • Models must use UUID primary keys for CRUD validation flow.
  • Primary key and foreign key fields are expected to define explicit db_column.
  • mo_crud_kit.read() requires queryset .values() input.
  • mo_crud_kit.delete() is not currently exposed.

Troubleshooting

  • read() fails with shape/type errors
    Confirm queryset input uses .values() and field names align with frame columns.
  • Rows are silently excluded from writes
    Inspect invalid_model_frms and the configured error column to trace validation failures.
  • FK validation fails unexpectedly
    Check UUID types and explicit db_column configuration on related fields.