You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Qdrant connector docs: VectorSchema(dtype=np.float32) (the class) fails to reload on the second run #2535
The first cocoindex update succeeds. The second run fails while CocoIndex reconciles its persisted collection-tracking record:
cocoindex._internal.serde.DeserializationError: Failed to deserialize msgspec payload
(type_hint=cocoindex.connectorkits.statediff.MutualTrackingRecord[cocoindex.connectors.qdrant._target._CollectionTrackingRecordCore],
source=prev_possible_records param of cocoindex.connectors.qdrant._target._CollectionHandler.reconcile())
_pickle.UnpicklingError: Forbidden global during unpickling: numpy.float32
Why
VectorSchema (a msgspec.Struct) declares dtype: np.dtype. np.float32 is a class, so msgspec reaches enc_hook, whose type is type → in _SERIALIZE_BY_PICKLE_TYPES → the value is pickled by reference to the global numpy.float32. On reload, _RestrictedUnpickler only allows ("numpy", "dtype"), numpy.ndarray, the registered dtype subclasses and _frombuffer — not the numpy.float32 scalar type class — so it raises.
Passing an instance (np.dtype(np.float32) or np.dtype("float32")) round-trips fine, because ("numpy", "dtype") is allow-listed.
To Reproduce
# run 1: OK, run 2: DeserializationErrorimportnumpyasnp, cocoindexascocofromcocoindex.connectorsimportqdrantfromcocoindex.resources.schemaimportVectorSchemaschema=awaitqdrant.CollectionSchema.create(
vectors={"v": qdrant.QdrantVectorDef(schema=VectorSchema(dtype=np.float32, size=384))}
)
awaitqdrant.mount_collection_target(DB, "coll", schema)
Minimal check of the round-trip, no indexing needed:
Either the docs use np.dtype(np.float32), or (better) VectorSchema normalizes/coerces dtype to a np.dtype instance (e.g. np.dtype(dtype)) so the documented class form works and is persisted safely.
Use VectorSchema(dtype=np.dtype(np.float32), size=...) (or VectorSchema(dtype=np.dtype("float32"), ...)).
Possibly related to #2011 (different root cause: numpy 1.x _frombuffer module path), but this one reproduces on numpy 2.x and is about the dtype argument form.
Activity
fix(schema): normalize VectorSchema dtype to np.dtype instance
fix(test): import io in TestNumpyScalarTypeSerde and format with ruff (…
fix(serde): recover run-1 records with numpy scalar classes; fail fas…
fix(schema): coerce VectorSchema dtype to np.dtype and allow numpy sc…
fix(test): import io in TestNumpyScalarTypeSerde and format with ruff (…
fix(schema): coerce VectorSchema dtype to np.dtype and allow numpy sc…
fix(test): import io in TestNumpyScalarTypeSerde and format with ruff (…