model after running run_pretraining.py script

[KavyaGujjala]

Hi,
I have run the code run_pretraining.py script on my domain specific data.

It seems like only checkpoints are saved. I have got two files 0000020.params and 0000020.states.

How can I save the model or get a model from .params and .states files in checkpoint folder so that I can use that model to get contextual embeddings.

Can someone please help me with this?

[eric-haibin-lin]

Hi @KavyaGujjala

By contextual embedding did you mean BertEmbedding in scripts/bert/embedding.py? You can modify this script with changes in https://github.com/dmlc/gluon-nlp/pull/640/files contributed by @Deseaus (which uses the load_parameters API to load the checkpoint file xx.params)

        if params_path:
            logger.info('Loading params from %s', params_path)
            self.bert.load_parameters(params_path, ctx=ctx, ignore_extra=True)

[KavyaGujjala]

Hi @KavyaGujjala

By contextual embedding did you mean BertEmbedding in scripts/bert/embedding.py? You can modify this script with changes in https://github.com/dmlc/gluon-nlp/pull/640/files contributed by @Deseaus (which uses the load_parameters API to load the checkpoint file xx.params)

        if params_path:
            logger.info('Loading params from %s', params_path)
            self.bert.load_parameters(params_path, ctx=ctx, ignore_extra=True)

Hey, Thanks a lot!!! This worked for me :)

Also bert returns only word embeddings. Do you have any idea on how to get meaningful sentence representations?

[szha]

@KavyaGujjala I see that this question is being tracked in #645 (thanks for opening that). I will close this issue now.