{"id":1941,"date":"2019-05-15T19:27:29","date_gmt":"2019-05-15T19:27:29","guid":{"rendered":"https:\/\/www.codeastar.com\/?p=1941"},"modified":"2019-05-15T19:27:42","modified_gmt":"2019-05-15T19:27:42","slug":"recurrent-neural-network-rnn-in-nlp-and-python-part-2","status":"publish","type":"post","link":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/","title":{"rendered":"RNN (Recurrent Neural Network) in NLP and Python – Part 2"},"content":{"rendered":"\n

From our Part 1<\/a> of NLP and Python topic, we talked about word pre-processing for a machine to handle words. This time, we are going to talk about building a model for a machine to classify words. We learned to use CNN to classify images<\/a> in past. Then we use another neural network, Recurrent Neural Network (RNN), to classify words now.<\/p>\n\n\n\n\n\n\n\n

What is Recurrent Neural Network (RNN)? <\/h3>\n\n\n\n

RNN is a class of deep neural networks and so is the CNN. Then what is the major difference between CNN and RNN? The spelling. (okay, don’t laugh, I’m serious :]] ) The “R” of RNN stands for Recurrent. It means process is occupied repeatedly and this is the feature we don’t see in CNN.<\/p>\n\n\n\n

In CNN, we call it a feed-forward network. While the input of layer 2 is the output of layer 1, the input of layer 3 is the output of layer 2 and the list goes on. But in RNN, things go repeatedly, as the inputs of layer 2 are<\/strong> the output of layer 1 and<\/strong> also the output of layer 2. A RNN not only produces output, it can copy and loop it back to the network. It turns out RNN is a neural network with memory.<\/p>\n\n\n\n

\"RNN,<\/figure><\/div>\n\n\n\n

It also makes RNN strong on handling sequence of data to predict precise outcome. The content of sequential data, e.g. speech, depends on how data is connected. When we have “Have”, “a” and “nice” as inputs, RNN remembers those inputs and predict “day” as output. That is also why RNN is widely used on text recognition and translation applications. We can explain a RNN with following diagram:<\/p>\n\n\n\n

\"Inside
(image source: http:\/\/colah.github.io\/posts\/2015-08-Understanding-LSTMs\/<\/a>)<\/figcaption><\/figure><\/div>\n\n\n\n

First, an input, X_t,<\/em> passes through RNN, A<\/em>. It starts from the first round. We call the first chunk of input as X_0<\/em>. RNN then produces hidden output h_0<\/em>. Then we go for the next round with input X_1<\/em>, h_0<\/em> is added to the RNN, and we have hidden output h_1<\/em>. The flow goes again and again until we put all our input into A<\/em>. Finally, we have h_t<\/em> as our output which trains with previous inputs. <\/p>\n\n\n\n

RNN in Python<\/h3>\n\n\n\n

From our Python Image Recognizer<\/a> post, we built a CNN model for image classification with Keras<\/a>. This time, we are going to use the Keras library again, but for a RNN model. Firstly, let’s import required modules.<\/p>\n\n\n\n

from keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Embedding, Input, Dense, CuDNNLSTM, concatenate, Bidirectional, SpatialDropout1D, Conv1D, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.optimizers import Adam\nfrom keras.models import Model\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\nimport keras.backend as K\nfrom sklearn.model_selection import KFold\n<\/pre>\n\n\n\n
Then we apply the word pre-processing function, punct_apo_fix<\/em>, from Part 1 and pre-process our training and testing data. <\/p>\n\n\n\n
df_merge = pd.concat([df_train[['id','comment_text']], df_test], axis=0)\ndf_merge[\"comment_text\"] = df_merge[\"comment_text\"].apply(lambda x: punct_apo_fix(x))\ndf_train_comment = df_merge.iloc[:df_train.shape[0],:]\ndf_test_comment = df_merge.iloc[df_train.shape[0]:,:]\ndf_train_comment = pd.concat([df_train_comment,df_train[['target']]],axis=1)\n<\/pre>\n\n\n\n
Now we normalize our training data and get the word vector indexes from the pre-trained fastText model.<\/p>\n\n\n\n
df_train_comment['target'] = np.where(df_train_comment['target'] >= 0.5, True, False)\ntokenizer = Tokenizer(num_words=100000)\ntokenizer.fit_on_texts(list(df_train_comment['comment_text']) + list(df_test_comment['comment_text']))\ntotal_unique_word = len(tokenizer.word_index) + 1\nwordvectors_index = KeyedVectors.load_word2vec_format(fasttext_300d_2m_model)\n<\/pre>\n\n\n\n
Next, we apply the fastText word vector indexes into words found from our training and testing data.<\/p>\n\n\n\n
EMBEDDINGS_DIMENSION = 300\nembedding_matrix = np.zeros((total_unique_word,EMBEDDINGS_DIMENSION))\nfor word, i in tokenizer.word_index.items():\n    if word in wordvectors_index.vocab:\n        embedding_matrix[i] = wordvectors_index[word]\n<\/pre>\n\n\n\n
Okay, we are going to the fun part of this project — build the model!<\/p>\n\n\n\n
RNN model with LSTM and Bidirectional Structure<\/h3>\n\n\n\n
Before we start to build our model, there are 2 techniques we can apply on RNN to make good the model. They are Long Short-Term Memory<\/a> (LSTM) and Bidirectional<\/a> RNN. Let’ start with LSTM first.<\/p>\n\n\n\n
When our input data is “People in Japan speak…<\/em> “. We can expect the output should be “Japanese<\/em>” from our human’s mind. But from a machine’s perspective, it does not have enough information to generate the output. It needs the relationship of “Japan” and “Japanese” from other inputs. That is why we need LSTM to extend RNN memory for not only current input but also previous inputs. <\/p>\n\n\n\n
Then we go for the Bidirectional RNN. The concept of Bidirectional structure is straight-forward. It duplicates a recurrent layer but in reverse order. When we have “Have a nice day<\/em>.” as input, it will turn out becoming 2 layers with “Have<\/em>“, “a<\/em>“, “nice<\/em>“, “day<\/em>“, “.<\/em>” and “.<\/em>“, “day<\/em>“, “nice<\/em>“, “a<\/em>“, “Have<\/em>“. So what is the benefit of using Bidirectional structure? Let’s think about “We go to a __________ to have lunch there<\/em>“. The RNN reads “We<\/em>“, “go<\/em>” “to<\/em>” forwardly and “there<\/em>“, “lunch<\/em>“, “have<\/em>” backwardly. Then it can predict “restaurant<\/em>” by relating inputs from the 2 layers.<\/p>\n\n\n\n
Now we put those techniques into our model:<\/p>\n\n\n\n
MAX_SEQUENCE_LENGTH = 256\ndef build_model(total_unique_word, embedding_matrix):\n    sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n    embedding_layer = Embedding(total_unique_word,\n                            EMBEDDINGS_DIMENSION,\n                            weights=[embedding_matrix],\n                            input_length=MAX_SEQUENCE_LENGTH,\n                            trainable=False)\n    x = embedding_layer(sequence_input)\n    x = SpatialDropout1D(0.2)(x)\n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)   \n    x = Conv1D(64, kernel_size = 2, padding = \"valid\", kernel_initializer = \"he_uniform\")(x)\n    avg_pool1 = GlobalAveragePooling1D()(x)\n    max_pool1 = GlobalMaxPooling1D()(x)     \n    x = concatenate([avg_pool1, max_pool1])\n    preds = Dense(1, activation='sigmoid')(x)\n    model = Model(sequence_input, preds)\n    model.compile(loss='binary_crossentropy',\n              optimizer=Adam(),\n              metrics=['acc'])\n    return model\n<\/pre>\n\n\n\n
As we mentioned in Part 1, a machine handles words using word vectors. That is why we apply Embedding<\/em> layer to convert input to vector. And you may notice that, instead of using Droupout <\/em>layer, we use SpatialDropout1D<\/em>. It will drop entire 1D feature maps, and make bigger difference for machine to learn.<\/p>\n\n\n\n
Predict Comment Classification<\/h3>\n\n\n\n
Data pre-processing, check. Model building, check. Now it is the time we go for predicting those comments are toxic or not. We also apply 3-fold cross validation<\/a> to enhance our prediction.<\/p>\n\n\n\n
train_text = pad_sequences(tokenizer.texts_to_sequences(df_train_comment[\"comment\"]), maxlen=MAX_SEQUENCE_LENGTH)\ntest_text = pad_sequences(tokenizer.texts_to_sequences(df_test_comment[\"comment\"]), maxlen=MAX_SEQUENCE_LENGTH)\ntrain_target = df_train_comment[\"target\"]\nn_splits=3\nsplits = list(KFold(n_splits).split(train_text,train_target))\ntest_preds = np.zeros((df_test_comment.shape[0]))\nfor fold in list(range(n_splits)):\n    K.clear_session()\n    tr_ind, val_ind = splits[fold]\n    checkpoint = ModelCheckpoint(f'gru_{fold}.hdf5', save_best_only = True)\n    earlystop = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=3)\n    model = build_model()\n    model.fit(train_text[tr_ind],\n        train_target[tr_ind],\n        batch_size=2048,\n        epochs=100,\n        validation_data=(train_text[val_ind], train_target[val_ind]),\n        callbacks = [earlystop,checkpoint])\n    test_preds += model.predict(test_text)[:,0]\ntest_preds \/= n_splits\nsubmission = pd.read_csv('..\/input\/jigsaw-unintended-bias-in-toxicity-classification\/sample_submission.csv', index_col='id')\nsubmission['prediction'] = test_preds\nsubmission.reset_index(drop=False, inplace=True)\n<\/pre>\n\n\n\n
After around 2 hours processing time, we have the prediction and we can discover how well a machine can do for a human’s work. <\/p>\n\n\n\n
validation_df = pd.merge(test_df, submission, on='id')\nvalidation_df[validation_df.prediction > 0.5].head()\nvalidation_df[validation_df.prediction < 0.5].head()\n<\/pre>\n\n\n\n
This is a group of machine classified toxic comments: <\/p>\n\n\n\n
\"Toxic<\/figure>\n\n\n\n
 And this is a group of non-toxic comments:  <\/p>\n\n\n\n
\"Non-toxic<\/figure>\n\n\n\n
I do not have any issue for non-toxic comments, just the toxic comments part is a bit, \"high moral standard\" :]] .<\/p>\n\n\n\n
When I submit above prediction to Kaggle, it turns out scoring 0.92x , i.e. 92.x% accuracy. There is still room for improvement, keep trying and learning!<\/p>\n\n\n\n
<\/div>\n\n\n\n
 What have we learnt in this post? <\/h3>\n\n\n\n
  1. Definition of Recurrent Neural Network<\/li>
  2. Concept of LSTM<\/li>
  3. Concept of Bidirectional structure<\/li>
  4. Building RNN model in Python<\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"
    From our Part 1 of NLP and Python topic, we talked about word pre-processing for a machine to handle words. This time, we are going to talk about building a model for a machine to classify words. We learned to use CNN to classify images in past. Then we use another neural network, Recurrent Neural […]<\/p>\n","protected":false},"author":1,"featured_media":1982,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"jetpack_publicize_message":"Build a RNN (Recurrent Neural Network) in Python","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[18],"tags":[57,26,146,140,145,144],"jetpack_publicize_connections":[],"yoast_head":"\nRNN (Recurrent Neural Network) in NLP and Python - Part 2 ⋆ Code A Star<\/title>\n<meta name=\"description\" content=\"In this post, we continue our journey in NLP. We will discuss using Recurrent Neural Network (RNN) with Python to classify comments from text source.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"RNN (Recurrent Neural Network) in NLP and Python - Part 2 ⋆ Code A Star\" \/>\n<meta property=\"og:description\" content=\"In this post, we continue our journey in NLP. We will discuss using Recurrent Neural Network (RNN) with Python to classify comments from text source.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Code A Star\" \/>\n<meta property=\"article:publisher\" content=\"codeastar\" \/>\n<meta property=\"article:author\" content=\"codeastar\" \/>\n<meta property=\"article:published_time\" content=\"2019-05-15T19:27:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-05-15T19:27:42+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.codeastar.com\/wp-content\/uploads\/2019\/05\/rnn.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"723\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Raven Hon\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@codeastar\" \/>\n<meta name=\"twitter:site\" content=\"@codeastar\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Raven Hon\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\"},\"author\":{\"name\":\"Raven Hon\",\"@id\":\"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd\"},\"headline\":\"RNN (Recurrent Neural Network) in NLP and Python – Part 2\",\"datePublished\":\"2019-05-15T19:27:29+00:00\",\"dateModified\":\"2019-05-15T19:27:42+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\"},\"wordCount\":876,\"commentCount\":3,\"publisher\":{\"@id\":\"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd\"},\"keywords\":[\"deep learning\",\"k-fold cross validation\",\"LSTM\",\"NLP\",\"Recurrent Neural Network\",\"RNN\"],\"articleSection\":[\"Learn Machine Learning\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\",\"url\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\",\"name\":\"RNN (Recurrent Neural Network) in NLP and Python - Part 2 ⋆ Code A Star\",\"isPartOf\":{\"@id\":\"https:\/\/www.codeastar.com\/#website\"},\"datePublished\":\"2019-05-15T19:27:29+00:00\",\"dateModified\":\"2019-05-15T19:27:42+00:00\",\"description\":\"In this post, we continue our journey in NLP. We will discuss using Recurrent Neural Network (RNN) with Python to classify comments from text source.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.codeastar.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"RNN (Recurrent Neural Network) in NLP and Python – Part 2\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.codeastar.com\/#website\",\"url\":\"https:\/\/www.codeastar.com\/\",\"name\":\"Code A Star\",\"description\":\"We don't wish upon a star, we code a star\",\"publisher\":{\"@id\":\"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.codeastar.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd\",\"name\":\"Raven Hon\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.codeastar.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/08\/logo70.png?fit=70%2C70&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/08\/logo70.png?fit=70%2C70&ssl=1\",\"width\":70,\"height\":70,\"caption\":\"Raven Hon\"},\"logo\":{\"@id\":\"https:\/\/www.codeastar.com\/#\/schema\/person\/image\/\"},\"description\":\"Raven Hon is\u00a0a 20 years+ veteran in information technology industry who has worked on various projects from console, web, game, banking and mobile applications in different sized companies.\",\"sameAs\":[\"https:\/\/www.codeastar.com\",\"codeastar\",\"https:\/\/twitter.com\/codeastar\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"RNN (Recurrent Neural Network) in NLP and Python - Part 2 ⋆ Code A Star","description":"In this post, we continue our journey in NLP. We will discuss using Recurrent Neural Network (RNN) with Python to classify comments from text source.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/","og_locale":"en_US","og_type":"article","og_title":"RNN (Recurrent Neural Network) in NLP and Python - Part 2 ⋆ Code A Star","og_description":"In this post, we continue our journey in NLP. We will discuss using Recurrent Neural Network (RNN) with Python to classify comments from text source.","og_url":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/","og_site_name":"Code A Star","article_publisher":"codeastar","article_author":"codeastar","article_published_time":"2019-05-15T19:27:29+00:00","article_modified_time":"2019-05-15T19:27:42+00:00","og_image":[{"width":1000,"height":723,"url":"https:\/\/www.codeastar.com\/wp-content\/uploads\/2019\/05\/rnn.png","type":"image\/png"}],"author":"Raven Hon","twitter_card":"summary_large_image","twitter_creator":"@codeastar","twitter_site":"@codeastar","twitter_misc":{"Written by":"Raven Hon","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#article","isPartOf":{"@id":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/"},"author":{"name":"Raven Hon","@id":"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd"},"headline":"RNN (Recurrent Neural Network) in NLP and Python – Part 2","datePublished":"2019-05-15T19:27:29+00:00","dateModified":"2019-05-15T19:27:42+00:00","mainEntityOfPage":{"@id":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/"},"wordCount":876,"commentCount":3,"publisher":{"@id":"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd"},"keywords":["deep learning","k-fold cross validation","LSTM","NLP","Recurrent Neural Network","RNN"],"articleSection":["Learn Machine Learning"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/","url":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/","name":"RNN (Recurrent Neural Network) in NLP and Python - Part 2 ⋆ Code A Star","isPartOf":{"@id":"https:\/\/www.codeastar.com\/#website"},"datePublished":"2019-05-15T19:27:29+00:00","dateModified":"2019-05-15T19:27:42+00:00","description":"In this post, we continue our journey in NLP. We will discuss using Recurrent Neural Network (RNN) with Python to classify comments from text source.","breadcrumb":{"@id":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.codeastar.com\/recurrent-neural-network-rnn-in-nlp-and-python-part-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.codeastar.com\/"},{"@type":"ListItem","position":2,"name":"RNN (Recurrent Neural Network) in NLP and Python – Part 2"}]},{"@type":"WebSite","@id":"https:\/\/www.codeastar.com\/#website","url":"https:\/\/www.codeastar.com\/","name":"Code A Star","description":"We don't wish upon a star, we code a star","publisher":{"@id":"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.codeastar.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.codeastar.com\/#\/schema\/person\/832d202eb92a3d430097e88c6d0550bd","name":"Raven Hon","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.codeastar.com\/#\/schema\/person\/image\/","url":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/08\/logo70.png?fit=70%2C70&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/08\/logo70.png?fit=70%2C70&ssl=1","width":70,"height":70,"caption":"Raven Hon"},"logo":{"@id":"https:\/\/www.codeastar.com\/#\/schema\/person\/image\/"},"description":"Raven Hon is\u00a0a 20 years+ veteran in information technology industry who has worked on various projects from console, web, game, banking and mobile applications in different sized companies.","sameAs":["https:\/\/www.codeastar.com","codeastar","https:\/\/twitter.com\/codeastar"]}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/05\/rnn.png?fit=1000%2C723&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8PcRO-vj","jetpack-related-posts":[{"id":764,"url":"https:\/\/www.codeastar.com\/convolutional-neural-network-python\/","url_meta":{"origin":1941,"position":0},"title":"Python Image Recognizer with Convolutional Neural Network","author":"Raven Hon","date":"February 11, 2018","format":false,"excerpt":"On our data science journey, we have solved classification and regression problems. What's next? There is one popular machine learning territory we have not set feet on yet --- the image recognition. But now the wait is over, in this post we are going to teach our machine to recognize\u2026","rel":"","context":"In "Learn Machine Learning"","block_context":{"text":"Learn Machine Learning","link":"https:\/\/www.codeastar.com\/category\/machine-learning\/"},"img":{"alt_text":"Teach our machine with Convolutional Neural Network","src":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/learning.png?fit=1052%2C744&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/learning.png?fit=1052%2C744&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/learning.png?fit=1052%2C744&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/learning.png?fit=1052%2C744&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/learning.png?fit=1052%2C744&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2134,"url":"https:\/\/www.codeastar.com\/nmt-make-an-easy-neural-machine-translator\/","url_meta":{"origin":1941,"position":1},"title":"NMT – make an easy Neural Machine Translator","author":"Raven Hon","date":"November 10, 2019","format":false,"excerpt":"I haven't updated this blog for a few months. As there is something big happened in my hometown. But life must go on, so we come back here and learn a new topic --- NMT (Neural Machine Translation). You may try the translate service before (if not, let's Google Translate),\u2026","rel":"","context":"In "Learn Machine Learning"","block_context":{"text":"Learn Machine Learning","link":"https:\/\/www.codeastar.com\/category\/machine-learning\/"},"img":{"alt_text":"NMT","src":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2020\/01\/nmt_4.png?fit=1052%2C551&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2020\/01\/nmt_4.png?fit=1052%2C551&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2020\/01\/nmt_4.png?fit=1052%2C551&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2020\/01\/nmt_4.png?fit=1052%2C551&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2020\/01\/nmt_4.png?fit=1052%2C551&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":815,"url":"https:\/\/www.codeastar.com\/visualize-convolutional-neural-network\/","url_meta":{"origin":1941,"position":2},"title":"Visualize a Convolutional Neural Network","author":"Raven Hon","date":"February 21, 2018","format":false,"excerpt":"On last post, we tried our image recognition project\u00a0with handwritten digits. We used a Convolutional Neural Network (CNN) to train our machine and it did pretty well with 99.47% accuracy. We learnt how a CNN works by actually implementing a model. Today, we move one step further to learn more\u2026","rel":"","context":"In "Learn Machine Learning"","block_context":{"text":"Learn Machine Learning","link":"https:\/\/www.codeastar.com\/category\/machine-learning\/"},"img":{"alt_text":"Visualize a CNN","src":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/cnn_seal.png?fit=1052%2C744&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/cnn_seal.png?fit=1052%2C744&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/cnn_seal.png?fit=1052%2C744&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/cnn_seal.png?fit=1052%2C744&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2018\/02\/cnn_seal.png?fit=1052%2C744&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":1895,"url":"https:\/\/www.codeastar.com\/word-embedding-in-nlp-and-python-part-1\/","url_meta":{"origin":1941,"position":3},"title":"Word Embedding in NLP and Python – Part 1","author":"Raven Hon","date":"April 30, 2019","format":false,"excerpt":"We have handled text in machine learning using TFIDF. And we can use it to build word cloud for analytic purpose. But is it the capability of a machine can do on text? Definitely not, as we just haven't let machine to \"learn\" about text yet. TFIDF is a statistics\u2026","rel":"","context":"In "Learn Machine Learning"","block_context":{"text":"Learn Machine Learning","link":"https:\/\/www.codeastar.com\/category\/machine-learning\/"},"img":{"alt_text":"Happy word embedding","src":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/04\/happy_embedding.png?fit=800%2C779&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/04\/happy_embedding.png?fit=800%2C779&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/04\/happy_embedding.png?fit=800%2C779&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/04\/happy_embedding.png?fit=800%2C779&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2027,"url":"https:\/\/www.codeastar.com\/save-and-load-your-rnn-model\/","url_meta":{"origin":1941,"position":4},"title":"Save and Load your RNN model","author":"Raven Hon","date":"June 25, 2019","format":false,"excerpt":"In this blog, we tasted different kinds of machine learning projects so far. Our projects included prediction on stock price, image recognizer on hand writing, NLP on comment classification and others. There was one thing in common --- we used long time to train a model. It is okay to\u2026","rel":"","context":"In "Learn Machine Learning"","block_context":{"text":"Learn Machine Learning","link":"https:\/\/www.codeastar.com\/category\/machine-learning\/"},"img":{"alt_text":"Save and Load Model","src":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/06\/save_n_load.png?fit=1100%2C400&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/06\/save_n_load.png?fit=1100%2C400&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/06\/save_n_load.png?fit=1100%2C400&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/06\/save_n_load.png?fit=1100%2C400&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2019\/06\/save_n_load.png?fit=1100%2C400&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2404,"url":"https:\/\/www.codeastar.com\/the-lazy-and-easy-pre-trained-translator-of-the-year\/","url_meta":{"origin":1941,"position":5},"title":"The Lazy (and Easy) Pre-Trained Translator of the Year","author":"Raven Hon","date":"November 30, 2022","format":false,"excerpt":"We made our own Neural Machine Translator (NMT) in 2019, which helped us to translate Dutch to English. Now it is 2022, and many things have changed in the world of Data Science. The arrival of Bidirectional Encoder Representations from Transformers (BERT), a pre-trained transformer model, in 2019 brought a\u2026","rel":"","context":"In "Learn Machine Learning"","block_context":{"text":"Learn Machine Learning","link":"https:\/\/www.codeastar.com\/category\/machine-learning\/"},"img":{"alt_text":"Lazy Translator","src":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2022\/11\/lazy_translator.png?fit=1200%2C469&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2022\/11\/lazy_translator.png?fit=1200%2C469&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2022\/11\/lazy_translator.png?fit=1200%2C469&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2022\/11\/lazy_translator.png?fit=1200%2C469&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.codeastar.com\/wp-content\/uploads\/2022\/11\/lazy_translator.png?fit=1200%2C469&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/posts\/1941"}],"collection":[{"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/comments?post=1941"}],"version-history":[{"count":33,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/posts\/1941\/revisions"}],"predecessor-version":[{"id":1983,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/posts\/1941\/revisions\/1983"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/media\/1982"}],"wp:attachment":[{"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/media?parent=1941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/categories?post=1941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.codeastar.com\/wp-json\/wp\/v2\/tags?post=1941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}