Text Generation using RNN & GPT2

Amber Ivanna Trujillo
7 min readOct 30, 2023

Understanding Language

Language is one of the most complex aspects of our existence. We use language to communicate our thoughts and choices. Every language is defined with a list of characters called the alphabet, a vocabulary and a set of rules called grammar. Yet it is not a trivial task to understand and learn a language.

It takes years for a human to grasp a few languages, let alone become a master of it. Most languages have vast vocabularies which are ever expanding along with complex and fuzzy grammatical rules and structures. We see/think and write/share differently. To put simply, languages are ambiguous. The ambiguity, sarcasm, context, and continuous changes to usage patterns are some of the challenges associated with language.

Examples of ambiguity

Ambiguity, generally used in natural language processing, can be referred as the ability of being understood in more than one way. In simple terms, we can say that ambiguity is the capability of being understood in more than one way. Natural language is very ambiguous. NLP has the following types of ambiguities

  • The bowler made the batsman duck.(The word duck, is it verb or noun?)
  • Stolen painting found by tree.(The word tree, is it an actor or a place?)
  • Call me taxi — This statement might be asking the listener to literally say that the speaker is a taxi or could be used to tell someone to summon a taxi.
  • Are you comfortable with python? — — The word python, is it representing the programming language python or a snake or something else?

Text Generation Why Recurrent Neural Networks?

Recurrent neural networks were created because there were a few issues in the feed-forward neural network:

  • Cannot handle sequential data
  • Considers only the current input
  • Cannot memorize previous inputs

The solution to these issues is the Recurrent Neural Network (RNN). An RNN can handle sequential data, accepting the current input data, and previously received inputs. RNNs can memorize previous inputs due to their internal memory

When sequence of information is important we are using RNN. Recurrent Neural Networks or RNNs are a class of networks that allow previous outputsto be used as…

--

--

Amber Ivanna Trujillo

I write about Technical stuff, interview questions, and finance Deep Learning, LLM, Startup, Influencer, Executive, Data Science Manager