Skip to content

Qdrant connector docs: VectorSchema(dtype=np.float32) (the class) fails to reload on the second run #2535

Description

@james-gibson

Describe the bug

The Qdrant connector docs pass the np.float32 class (not a np.dtype instance) to VectorSchema:

from cocoindex.resources.schema import VectorSchema
import numpy as np

schema = await qdrant.CollectionSchema.create(
    vectors={
        "text": qdrant.QdrantVectorDef(schema=VectorSchema(dtype=np.float32, size=384), distance="cosine"),
        ...
    }
)

— see the Named vectors and Dense + sparse vectors sections of https://cocoindex.io/docs/connectors/qdrant/ (source: docs/src/content/docs/connectors/qdrant.mdx).

The first cocoindex update succeeds. The second run fails while CocoIndex reconciles its persisted collection-tracking record:

cocoindex._internal.serde.DeserializationError: Failed to deserialize msgspec payload
  (type_hint=cocoindex.connectorkits.statediff.MutualTrackingRecord[cocoindex.connectors.qdrant._target._CollectionTrackingRecordCore],
   source=prev_possible_records param of cocoindex.connectors.qdrant._target._CollectionHandler.reconcile())
_pickle.UnpicklingError: Forbidden global during unpickling: numpy.float32

Why

VectorSchema (a msgspec.Struct) declares dtype: np.dtype. np.float32 is a class, so msgspec reaches enc_hook, whose type is type → in _SERIALIZE_BY_PICKLE_TYPES → the value is pickled by reference to the global numpy.float32. On reload, _RestrictedUnpickler only allows ("numpy", "dtype"), numpy.ndarray, the registered dtype subclasses and _frombuffer — not the numpy.float32 scalar type class — so it raises.

Passing an instance (np.dtype(np.float32) or np.dtype("float32")) round-trips fine, because ("numpy", "dtype") is allow-listed.

To Reproduce

# run 1: OK, run 2: DeserializationError
import numpy as np, cocoindex as coco
from cocoindex.connectors import qdrant
from cocoindex.resources.schema import VectorSchema

schema = await qdrant.CollectionSchema.create(
    vectors={"v": qdrant.QdrantVectorDef(schema=VectorSchema(dtype=np.float32, size=384))}
)
await qdrant.mount_collection_target(DB, "coll", schema)

Minimal check of the round-trip, no indexing needed:

import io, pickle, numpy as np
from cocoindex._internal.serde import _RestrictedUnpickler
_RestrictedUnpickler(io.BytesIO(pickle.dumps(np.float32, 5))).load()        # UnpicklingError: numpy.float32
_RestrictedUnpickler(io.BytesIO(pickle.dumps(np.dtype(np.float32), 5))).load()  # OK -> float32

Expected behavior

Either the docs use np.dtype(np.float32), or (better) VectorSchema normalizes/coerces dtype to a np.dtype instance (e.g. np.dtype(dtype)) so the documented class form works and is persisted safely.

Versions

  • cocoindex==1.0.25, qdrant-client==1.16.x, numpy 2.x
  • macOS arm64, Python 3.12 (uv)

Workaround

Use VectorSchema(dtype=np.dtype(np.float32), size=...) (or VectorSchema(dtype=np.dtype("float32"), ...)).

Possibly related to #2011 (different root cause: numpy 1.x _frombuffer module path), but this one reproduces on numpy 2.x and is about the dtype argument form.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions