Skip to content

Data Operations (CRUD)

Use mo_crud_kit for model-aware bulk create/read/update workflows with Polars frames. This workflow is designed for data-heavy endpoints where serializer-by-row patterns become a bottleneck. It applies directly to high-volume ingestion and update APIs.

When to use mo_crud_kit — and when not to

mo_crud_kit is designed for workflows where data arrives or is consumed in bulk: ingestion pipelines, large exports, high-volume upserts, or any operation where the dataset is large enough that row-by-row serializer overhead or bulk_create become the bottleneck. It works on Polars DataFrame and LazyFrame inputs — not on individual model instances.

For everyday operations — creating a single record, updating a user profile, returning a list of 20 results — use native Django serializers. They are simpler, easier to debug, and the right tool at that scale. Reach for mo_crud_kit when volume is the problem. The two approaches are complementary; most projects use both.

Prerequisites

  • Models are migrated and uses UUID primary/foreign keys.

Implementation

mo_crud_kit provides model-aware operations for bulk create, read, and update. It uses model metadata plus Polars validation to keep writes fast and consistent.

1. Model Frames

mo_crud_kit works on a model-frame mapping:

{
    OrderModel: order_df_or_lazy,
    OrderItemModel: item_df_or_lazy,
}

Each write operation returns:

  • status: ok, partial_ok, or fail
  • valid_model_frms: rows that passed validation
  • invalid_model_frms: rows with error metadata

Sample Model Frame

import polars as pl
from apps.orders.models import OrderModel

order_df = pl.DataFrame(
    {
        "order_id": ["f2fa1a5b7abf4f37a3f7e14725c0b211"],
        "status": ["draft"],
        "total_amount": [120.50],
    }
)

model_frms = {
    OrderModel: order_df,
}

2. Create

Create rows from model-to-frame mappings with optional validation pipeline.

Usage:

from django_mindoff import mo_crud_kit

status, valid_model_frms, invalid_model_frms = mo_crud_kit.create(
    {
        OrderModel: order_df,
        OrderItemModel: order_item_df,
    },
    is_partial=False,
    validation_level="full",
    batch_size=1000,
)

Parameters:

  • model_frms (dict[type[models.Model], pl.DataFrame|pl.LazyFrame]): Input model-frame mapping for bulk insert.
  • using (str|BaseDatabaseWrapper|None, default=None): Which database to write to. None targets the default database and leaves database routing untouched. Pass an alias to target another configured database, or a live Django connection object to target one provisioned at runtime whose credentials were never written into settings.DATABASES. See the note on dynamic databases below.
  • is_partial (bool, default=False): If True, allows partial save when only a subset of rows are valid (applies to validation_level="full" only).
  • validation_level ("full"|"columns_only"|"none", default="full"): Selects how much validation runs before the insert.
  • batch_size (int, default=1000): Maximum number of rows written per INSERT batch. Has no meaning when is_validate_only=True: nothing is batched because nothing is written.
  • is_validate_only (bool, default=False): If True, runs the validation pipeline and returns its verdict without writing anything — a preview of what a real create() would classify.

Varieties:

  • validation_level="full" (default): runs ColumnValidator -> RowValidator -> ForeignKeyValidator and honors is_partial.
  • validation_level="columns_only": runs only ColumnValidator to normalize frame shape (rename to db_column, add missing/auto columns, drop extras) and writes directly, trusting the caller's row values and foreign keys. Row-level passes are skipped, so field defaults, auto_now/auto_now_add timestamps, UUID generation, and type coercion are NOT applied — the caller must supply database-ready values for every required column. is_partial does not apply (no per-row invalidation).
  • validation_level="none": skips all validation (including column normalization) and writes the frames as-is; emits an unsafe-write warning.
  • Partial-save mode (full only): is_partial=False fails if any invalid rows exist; is_partial=True saves valid rows and returns invalid rows separately.
  • is_validate_only=True: a dry run at any validation_level. The selected validation runs exactly as it would for a real create — including the foreign-key existence queries at "full", which are reads — and then the write is skipped entirely. Nothing is written even at "none" or "columns_only", which otherwise write directly.

Possible responses:

  • Returns ("ok", valid_model_frms, {}) when all rows are valid and inserted.
  • Returns ("partial_ok", valid_model_frms, invalid_model_frms) when partial mode is enabled and some rows are invalid.
  • Returns ("fail", valid_model_frms, invalid_model_frms) when validation fails and no write should proceed.

Notes:

  • Invalid rows contain an error column (POLARS_VALIDATOR_ERROR_COL or __error__info).
  • An unresolvable foreign key is an invalid row like any other: the row is marked in the error column and returned in invalid_model_frms, so is_partial=True writes the rest of the batch and is_partial=False fails the call without writing. It does not raise.
  • is_validate_only=True returns the same (status, valid_model_frms, invalid_model_frms) shape as a real create, so it is a drop-in preview of one. What it cannot report is anything the write itself would raise: the database is never contacted for the write, so integrity errors, unsupported-backend errors, and connection failures surface only on the real call.
  • validation_level="none" still emits its unsafe-write warning under is_validate_only=True. Nothing is written, so the warning is not about this call — it is the only signal that the preview checked nothing at all, and therefore says nothing about the write it stands in for.
  • validation_level="none" skips safety checks and may persist unsafe data.
  • Rows are written with each backend's fastest same-transaction bulk loader — PostgreSQL COPY, MySQL LOAD DATA LOCAL INFILE (auto-falling back to row binding when the server forbids it), SQLite executemany — so the columnar frame reaches the database without a per-row Python detour, and the whole create stays atomic.
  • Dynamic databases: a connection passed as using does not have to appear in settings.DATABASES, but foreign-key validation issues ORM queries, so at validation_level="full" the connection must also be registered in django.db.connections under its alias. If it is not, use validation_level="columns_only".

3. Read

Read queryset data into Polars with streaming/pagination variants.

Usage:

from django_mindoff import mo_crud_kit

frm, stats = mo_crud_kit.read(
    OrderModel.objects.filter(is_active=True).values(),
    page_number=1,
    is_lazy=False,
    batch_size=100,
)

Parameters:

  • qs (models.QuerySet): Queryset that must use .values() output.
  • page_number (int|None, default=None): When provided, enables pagination mode. When None, uses streaming mode.
  • is_lazy (bool, default=False): If True, returns pl.LazyFrame; otherwise returns pl.DataFrame. In streaming mode with the auto/connectorx engines this is a real larger-than-RAM scan: rows are streamed to a temporary Parquet file and the returned LazyFrame scans it lazily (the temp file is removed when the frame is garbage-collected). The iterator engine keeps the legacy deferred-.lazy() behavior.
  • batch_size (int, default=0): Chunk/page size. Auto-resolved when 0. Drives the lazy-scan/streaming fetch size and pagination; eager fast-path reads fetch in one transfer.
  • with_stats (bool, default=True): When True, computes total_count/total_pages (and the empty-set short-circuit) via extra exists()/count() queries. Set False for the fastest read: those queries are skipped, total_count/total_pages are None, and pagination derives has_next by fetching one extra row.
  • json_column_mode ("auto"|"object"|"text", default="auto"): How JSONField columns are returned. text keeps raw JSON text (Utf8); object parses to pl.Object; auto resolves to text.

Varieties:

  • Streaming mode (page_number=None): reads the full dataset.
  • Pagination mode (page_number=<n>): reads one page and returns paging metadata.
  • Materialization mode: eager (DataFrame) or lazy (LazyFrame).

Possible responses:

  • Returns (frm, stats) where frm is DataFrame/LazyFrame and stats includes: mode, batch_size, total_count, total_pages, current_page, has_next, has_previous.
  • Returns empty frame with zeroed stats for empty querysets.
  • Raises validation error if queryset is not .values()-based.

Note:

  • Frames are normalized to the canonical model dtypes (_crud_kit.dtypes.DJANGO_TO_POLARS_TYPE_MAP), the same mapping the create/update validators enforce, so a read can be fed straight back into update.

4. Read Batches

Stream a queryset as an iterator of Polars frames (memory-bounded).

Usage:

from django_mindoff import mo_crud_kit

for frm in mo_crud_kit.read_batches(
    OrderModel.objects.filter(is_active=True).values(),
    batch_size=10_000,
):
    process(frm)

Parameters:

  • qs (models.QuerySet): Queryset that must use .values() output.
  • batch_size (int, default=0): Rows per yielded frame. Auto-resolved to 1000 when 0.
  • json_column_mode ("auto"|"object"|"text", default="auto"): JSON handling, identical to read(). Each batch is streamed from Django's cursor and normalized to the canonical model dtypes.

Possible responses:

  • Returns a generator of pl.DataFrame chunks; never concatenated, so the full result set is never held in memory at once.
  • Raises validation error if queryset is not .values()-based.

5. Update

Upsert rows from model-to-frame mappings via a staged merge.

Usage:

from django_mindoff import mo_crud_kit

status, valid_model_frms, invalid_model_frms = mo_crud_kit.update(
    {
        OrderModel: order_updates_df,
    },
    is_partial=True,
    validation_level="full",
    batch_size=1000,
)

Parameters:

  • model_frms (dict[type[models.Model], pl.DataFrame|pl.LazyFrame]): Input model-frame mapping for bulk update/upsert.
  • using (str|BaseDatabaseWrapper|None, default=None): Which database to write to. None targets the default database and leaves database routing untouched. Pass an alias to target another configured database, or a live Django connection object to target one provisioned at runtime whose credentials were never written into settings.DATABASES. See the note on dynamic databases below.
  • is_partial (bool, default=False): If True, allows valid rows to proceed even when invalid rows exist (applies to validation_level="full" only).
  • validation_level ("full"|"columns_only"|"none", default="full"): Selects how much validation runs before the upsert.
  • batch_size (int, default=1000): Batch size used to load the staging table. Has no meaning when is_validate_only=True: there is no staging table to load because nothing is written.
  • is_validate_only (bool, default=False): If True, runs the validation pipeline and returns its verdict without writing anything — a preview of what a real update() would classify.
  • skip_db_fill (bool, default=False): Deprecated and inert; scheduled for removal in 1.0. It used to skip a prefetch SELECT that back-filled the columns a frame omitted. There is no longer a back-fill to skip — omitted columns are simply not written — so passing it changes nothing and emits a DeprecationWarning.

Varieties:

  • validation_level="full" (default): runs ColumnValidator -> RowValidator -> ForeignKeyValidator and honors is_partial.
  • validation_level="columns_only": runs only ColumnValidator and upserts directly, trusting the caller's row values and foreign keys. Row-level passes are skipped, so the caller must supply database-ready values.
  • validation_level="none": skips all validation; upserts as-is and emits an unsafe-write warning.
  • is_validate_only=True: a dry run at any validation_level. The selected validation runs exactly as it would for a real update — including the foreign-key existence queries at "full", which are reads — and then the upsert is skipped entirely. Nothing is written even at "none" or "columns_only", which otherwise write directly.

Possible responses:

  • Returns ("ok", valid_model_frms, {}) when all rows are valid and updated.
  • Returns ("partial_ok", valid_model_frms, invalid_model_frms) when partial mode is enabled and some rows are invalid.
  • Returns ("fail", valid_model_frms, invalid_model_frms) when validation blocks update.

Notes:

  • Only the columns your frame carries are written. A column the frame omits is left out of both the INSERT list and the SET clause, so an existing row keeps whatever it already holds and a new row takes the column's database default. Nothing is read back first, so two writers updating different columns of the same row no longer overwrite each other. The returned frames therefore carry exactly the columns that were written.
  • A frame carrying only the primary key has nothing to write; that is a no-op and emits a RuntimeWarning rather than failing silently.
  • Because update() is an upsert, a primary key that does not exist yet is an INSERT. If the frame omits a NOT NULL column that has no database default, that INSERT fails with the database's own integrity error and the whole call rolls back — use create() for genuinely new rows, or supply the column.
  • Invalid rows include model-aware error details in error column.
  • An unresolvable foreign key is an invalid row like any other: the row is marked in the error column and returned in invalid_model_frms, so is_partial=True merges the rest of the batch and is_partial=False fails the call without writing. It does not raise.
  • is_validate_only=True returns the same (status, valid_model_frms, invalid_model_frms) shape as a real update, so it is a drop-in preview of one. What it cannot report is anything the merge itself would surface: the database is never contacted for the write, so the PK-only no-op warning above, integrity errors from an upsert-insert, unsupported-backend errors, and connection failures appear only on the real call.
  • validation_level="none" still emits its unsafe-write warning under is_validate_only=True. Nothing is written, so the warning is not about this call — it is the only signal that the preview checked nothing at all, and therefore says nothing about the write it stands in for.
  • The staging merge loads the staging table with the same fast same-transaction bulk loader as create() (PostgreSQL COPY, MySQL LOAD DATA LOCAL INFILE, SQLite executemany) and then merges set-based in SQL, so no per-row Python materialization occurs.
  • Dynamic databases: a connection passed as using does not have to appear in settings.DATABASES. Foreign-key validation is the only stage that issues an ORM query, so either register the connection in django.db.connections under its alias or use validation_level="columns_only".

Example Usage

from django_mindoff import mo_crud_kit
from apps.orders.models import OrderModel

# CREATE (full validation, written in batch_size chunks via the backend's fast
# bulk loader: PostgreSQL COPY / MySQL LOAD DATA / SQLite executemany)
create_status, create_valid, create_invalid = mo_crud_kit.create(
    {OrderModel: order_df},
    validation_level="full",   # "full" | "columns_only" | "none"
    is_partial=False,
    batch_size=1000,
)

# CREATE (fast path: caller guarantees clean, DB-ready rows — skip row+FK passes)
mo_crud_kit.create(
    {OrderModel: order_df},
    validation_level="columns_only",
)

# DRY RUN (validate and classify, write nothing — same return shape as a real call)
preview_status, preview_valid, preview_invalid = mo_crud_kit.create(
    {OrderModel: order_df},
    validation_level="full",
    is_partial=True,
    is_validate_only=True,
)
# Nothing was written. Show the user preview_invalid, then commit the same frames
# by re-running without the flag. `update()` takes `is_validate_only` identically.

# READ (queryset must use values())
orders_frm, stats = mo_crud_kit.read(
    OrderModel.objects.filter(is_active=True).values(),
    page_number=1,
    batch_size=100,
    with_stats=False,   # skip exists()/count() for the fastest read
)

# LARGER-THAN-RAM: a real lazy scan (streamed to disk, scanned lazily)
lazy_frm, _ = mo_crud_kit.read(
    OrderModel.objects.all().values(), is_lazy=True
)

# LARGER-THAN-RAM: process in memory-bounded chunks (never concatenated)
for chunk in mo_crud_kit.read_batches(
    OrderModel.objects.all().values(), batch_size=10_000
):
    handle(chunk)

# UPDATE (staged merge: bulk-load staging table, then set-based SQL merge)
update_status, update_valid, update_invalid = mo_crud_kit.update(
    {OrderModel: order_df},
    validation_level="full",   # "full" | "columns_only" | "none"
    is_partial=True,
    # Only the columns `order_df` carries are written. Anything it omits keeps
    # the value already stored, so a partial frame is a partial update.
)

Core Concepts

1. Polars Serialization

mo_crud_kit uses model metadata plus Polars validators (ColumnValidator, RowValidator, ForeignKeyValidator) to sanitize and validate rows before DB writes. This is the intended replacement for serializer-driven bulk validation in data-heavy pipelines.

What this gives you:

  • Type normalization aligned with Django field definitions.
  • Constraint checks (required/nullability, choices, min/max, length, FK consistency).
  • Structured invalid-row capture in the configured error column (POLARS_VALIDATOR_ERROR_COL, default __error__info).

An unresolvable foreign key is captured the same way as any other bad value — the row is marked in the error column and returned in invalid_model_frms rather than raising. One dangling reference therefore no longer costs the whole batch: with is_partial=True the remaining rows are written, and with is_partial=False the call fails without writing anything.

Validate before write

mo_crud_kit is built for validated tabular data. If you choose to skip the inbuilt validation + serialization, cover request-level validation in API code before sending it to CRUD Kit.

2. Previewing a Write (is_validate_only)

create() and update() accept is_validate_only=True to run the validation pipeline and hand back the valid/invalid split without writing anything — a confirm-before-commit step, or a dry run against production data.

  • The return shape is identical to a real call, so the flag is a drop-in swap.
  • No write happens at any validation_level, including none and columns_only, which otherwise write directly.
  • Validation is the real thing: at validation_level="full" the foreign-key checks still query the database.
  • batch_size is irrelevant here — nothing is batched because nothing is written.

What a preview cannot tell you is anything only the write can raise: database integrity errors, connection failures, and the primary-key-only no-op warning all come from the write path. A clean preview means validation passed, not that the write will.

3. Limitations

  • ManyToManyField is not supported in row validation.
  • BinaryField is not supported in row validation.
  • Any Django field not mapped in row validator dtype map is unsupported.
  • Models must use UUID primary keys for CRUD validation flow.
  • Primary key and foreign key fields are expected to define explicit db_column.
  • mo_crud_kit.read() requires queryset .values() input.
  • mo_crud_kit.delete() is not currently exposed.

Troubleshooting

  • read() fails with shape/type errors
    Confirm queryset input uses .values() and field names align with frame columns.
  • Rows are silently excluded from writes
    Inspect invalid_model_frms and the configured error column to trace validation failures.
  • FK validation fails unexpectedly
    Check UUID types and explicit db_column configuration on related fields.