Continuous Data Ingestion without Azure AI Indexers

[svenfeld]

I have a server which serves PDF files and additional metadata. I'm curious about what would be the best way to build a "pipeline" using prepdocs to have continuous data ingestion without relying on something like Azure AI Indexers.

I’d love to hear some ideas on how to approach this.

[Poonkuzhali (poonkuzhali)]

I am currently working on a blob triggered Azure Function for a similar situation. The entire prepdocs logic is imported to the Azure Function.

[Fabian Wallwitz (cforce)]

Is there any Code to Share - iac(bicep), Packaging as function and what strategy is implemented in existing codebase to reuse?

[Pamela Fox (pamelafox)]

There's a repo that uses functions for ingestion, with an admin in Streamlit: https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator

For this repo, we've definitely talked about moving prepdocs to an Azure Function. I think we'd still use it with a Blob storage indexer, though, for easier triggering and scheduling. cc Matt (@mattgotteiner)

Poonkuzhali (@poonkuzhali) Are you using an indexer with your setup?

[Poonkuzhali (poonkuzhali)]

Pamela Fox (@pamelafox) No. I am not using indexer, since I am not sure if the indexer will work with deletion of documents from the indexes.

I am working on two functions for two scenarios - addition of documents and removal of documents. Both have the same prepdocs logic, but the DocumentAction is set to Add and Remove respectively for each function.

Not sure if this the right way to approach this.

[cloudhunnter]

Poonkuzhali (@poonkuzhali) could your share your functions code, maybe? I am also looking for going this way, so we keep the citations, but move away from manually triggered ingestion by prepdocs.ps1

[Pamela Fox (pamelafox)]

Another option is a container apps job, versus Azure Functions, but there's still the question of triggering and scheduling.