EXPLORING UNIVERSAL SENTENCE ENCODER MODEL (USE)

UNIVERSAL SENTENCE ENCODER MODEL
UNIVERSAL SENTENCE ENCODER MODEL

(Image source: https://arxiv.org/pdf/1804.07754.pdf)

  • 3.3 Natural language inference (NLI):

    is the task of determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.

UNIVERSAL SENTENCE ENCODER MODEL

(Image source: https://paperswithcode.com/task/natural-language-inference)

The intuition is that by doing such trainings, the model can capture the most informative features and discard noise from text. This output embedding in turn will be applied to wide variety of NLP tasks such as relatedness, clustering, paraphrase detection and text classification.

  • 4.Results of UNIVERSAL SENTENCE ENCODER MODEL

     

    To compare the performance between using transfer learning via pretrained word embedding and no using any transfer learning, Word Embedding Association Test (WEAT) was used. WEAT includes:

    The results are summarized on the table as follows:

    UNIVERSAL SENTENCE ENCODER MODEL

The results show that:

  • Transfer learning from the Transformer based sentence encoder usually performs as good or better than transfer learning from the DAN encoder
  • Models that make use of sentence-level transfer learning tend to perform better than models that only use word-level transfer.
  • The best performance on most tasks is obtained by models that make use of both sentence and word level transfer

References:

Glove

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation, https://nlp.stanford.edu/pubs/glove.pdf

Word2vec

Mikolov, Tomas, et al. Efficient Estimation of Word Representations in Vector Space. Jan. 2013. arxiv.org, https://arxiv.org/abs/1301.3781v3.

fasText

Bojanowski, Piotr, et al. Enriching Word Vectors with Subword Information. July 2016. arxiv.org, https://arxiv.org/abs/1607.04606v2.

Transformer

Vaswani, Ashish, et al. Attention Is All You Need. June 2017. arxiv.org, https://arxiv.org/abs/1706.03762v5.

DAN

Iyyer, Mohit, et al. “Deep Unordered Composition Rivals Syntactic Methods for Text Classification.” Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, 2015, pp. 1681–91. ACLWeb, doi:10.3115/v1/P15-1162

Hiring Data Scientist / Engineer

We are looking for Data Scientist and Engineer.
Please check our Career Page.

AI / Data Science Project

Please check about experiences for Data Science Project

Vietnam AI / Data Science Lab

Vietnam AI Lab

Please also visit Vietnam AI Lab