{"id":15546,"date":"2022-01-31T15:07:39","date_gmt":"2022-01-31T20:07:39","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=15546"},"modified":"2025-08-12T11:49:09","modified_gmt":"2025-08-12T15:49:09","slug":"tensorflow-pytorch-ner-models-darkshield","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/","title":{"rendered":"Using Tensorflow and PyTorch NER Models in DarkShield"},"content":{"rendered":"<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">Named entity recognition (NER) is a type of machine learning (ML) used to detect named entities within the grammatical context of unstructured text (documents). NER is needed to find things like people names and street addresses, since those do not conform to patterns, nor likely have a match to values in a defined list (lookup set).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Since many entities like people\u2019s names or addresses are personally identifiable information (PII), <\/span><a href=\"https:\/\/www.iri.com\/products\/darkshield\"><span style=\"font-weight: 400;\">IRI DarkShield<\/span><\/a><span style=\"font-weight: 400;\"> uses NER to find such data and mask it. While knowing just a person\u2019s name alone might not be too much of a risk, in combination with other sensitive data, the risk for that person to be the target of cyber criminals increases if the dataset were breached.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">DarkShield has supported the import and further training of <\/span><a href=\"https:\/\/www.youtube.com\/watch?v=JkQA4uYraH0&amp;t=212&amp;ab_channel=IRITheCoSortCo\"><span style=\"font-weight: 400;\">OpenNLP models<\/span><\/a><span style=\"font-weight: 400;\"> to find and mask named entities for years. New in the 2022 DarkShield <\/span><a href=\"http:\/\/tinyurl.com\/DarkShieldAPI\"><span style=\"font-weight: 400;\">RPC API<\/span><\/a><span style=\"font-weight: 400;\"> though is support for state-of-the-art <\/span><a href=\"https:\/\/hackr.io\/blog\/pytorch-vs-tensorflow\"><span style=\"font-weight: 400;\">Tensorflow and PyTorch<\/span><\/a><span style=\"font-weight: 400;\"> NER models. This is a significant enhancement over the first set of fast, but fewer NER models based on OpenNLP.<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15552 alignleft\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-logo.png\" alt=\"\" width=\"171\" height=\"143\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-logo.png 450w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-logo-300x250.png 300w\" sizes=\"(max-width: 171px) 100vw, 171px\" \/>Tensorflow and PyTorch support many more publicly available, pre-trained NER models, including those from sources such as the <\/span><a href=\"https:\/\/huggingface.co\/models\"><span style=\"font-weight: 400;\">Hugging Face model hub<\/span><\/a><span style=\"font-weight: 400;\">. There are NER models available in over 100 languages from that hub alone. Many of these models take advantage of the relatively new <\/span><i><span style=\"font-weight: 400;\">Transformer <\/span><\/i><span style=\"font-weight: 400;\">machine learning architecture for reduced training times and improved accuracy.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15551 alignright\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/pytorch-logo.png\" alt=\"\" width=\"366\" height=\"100\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/pytorch-logo.png 450w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/pytorch-logo-300x82.png 300w\" sizes=\"(max-width: 366px) 100vw, 366px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">To use the Transformers search matcher with the DarkShield API, we recommend a system with a GPU to speed up the inference of most models (usually by at least 20 times). The DarkShield API will automatically download the dependencies necessary for the Transformers matcher when the server starts.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Models are specified either as a local directory, or from the Hugging Face model hub. If specifying directly from the Hugging Face model hub, then the <\/span><i><span style=\"font-weight: 400;\">model_util.py<\/span><\/i><span style=\"font-weight: 400;\"> Python script from the <\/span><i><span style=\"font-weight: 400;\">plankton\/utils<\/span><\/i><span style=\"font-weight: 400;\"> folder should be run beforehand to download the model. This script can be given the arguments of the model and tokenizer name (often the same as the model name).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">I will demonstrate the detection of named entities in English, Turkish, and Japanese text using three different NER models, all available from the Hugging Face model hub.<\/span><\/p>\n<h6><b>English<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">This first example is actually available as an example in the DarkShield API\u2019s OpenAPI spec.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">At the <\/span><i><span style=\"font-weight: 400;\">\/docs<\/span><\/i><span style=\"font-weight: 400;\"> endpoint on the server and port on which the DarkShield API is hosted, there is an interactive example of using a pre-trained NER model to detect four different named entities within a sentence.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15565 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformers-ner-context-create.png\" alt=\"\" width=\"400\" height=\"477\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformers-ner-context-create.png 591w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformers-ner-context-create-251x300.png 251w\" sizes=\"(max-width: 400px) 100vw, 400px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">First, a search context must be created. This can be done by trying out and executing a request to create a search context with a PyTorch named entity recognition model from the <\/span><i><span style=\"font-weight: 400;\">api\/darkshield\/searchContext.create<\/span><\/i><span style=\"font-weight: 400;\"> endpoint examples.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This example has no model or tokenizer specified, so it uses a default pretrained Bidirectional Encoder Representations from Transformers (BERT) model. This model should be downloaded beforehand by running the <\/span><i><span style=\"font-weight: 400;\">model_util.py<\/span><\/i><span style=\"font-weight: 400;\"> script available in the <\/span><i><span style=\"font-weight: 400;\">utils <\/span><\/i><span style=\"font-weight: 400;\">directory of the DarkShield API distribution with no arguments.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This is done by changing directory to the <\/span><i><span style=\"font-weight: 400;\">utils <\/span><\/i><span style=\"font-weight: 400;\">directory and running the command <\/span><i><span style=\"font-weight: 400;\">python model_util.py<\/span><\/i><span style=\"font-weight: 400;\">, assuming that python is on the PATH and can be referenced with the <\/span><i><span style=\"font-weight: 400;\">python<\/span><\/i><span style=\"font-weight: 400;\"> command. In some cases, <\/span><i><span style=\"font-weight: 400;\">python3<\/span><\/i><span style=\"font-weight: 400;\"> may need to be used instead of <\/span><i><span style=\"font-weight: 400;\">python<\/span><\/i><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15566 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformer-search-context-search.png\" alt=\"\" width=\"550\" height=\"378\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformer-search-context-search.png 648w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformer-search-context-search-300x206.png 300w\" sizes=\"(max-width: 550px) 100vw, 550px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Once the model has been successfully downloaded and a search context named <\/span><i><span style=\"font-weight: 400;\">TransformersMatcherContext <\/span><\/i><span style=\"font-weight: 400;\">created, the context can be used to search English text for named entities. From the <\/span><i><span style=\"font-weight: 400;\">api\/darkshield\/searchContext.search<\/span><\/i><span style=\"font-weight: 400;\"> examples, try out and execute the example to <\/span><i><span style=\"font-weight: 400;\">Search for named entities using the TransformersMatcherContext<\/span><\/i><span style=\"font-weight: 400;\">. The first time the example runs it will take more time than subsequent searches, as the model has to be initialized the first time.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15567 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformer-search-results.png\" alt=\"\" width=\"307\" height=\"433\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformer-search-results.png 307w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-transformer-search-results-213x300.png 213w\" sizes=\"(max-width: 307px) 100vw, 307px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">After sending the request to the API to search the text with this pretrained NER model, search results are returned that contain what named entities were found in the search, along with the label for the type of named entity the match is, and the probability that it is a named entity.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Note that various different types of named entities have been found, as the model was trained to label more than just people\u2019s names. John Doe was found as a PER entity, which means a person\u2019s name. New York City was found as a LOC entity, which means a location. \u2018Knicks\u2019 was found as an ORG entity, which means it was detected as an organization.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The DarkShield API offers flexibility with the <\/span><i><span style=\"font-weight: 400;\">entityLabels<\/span><\/i><span style=\"font-weight: 400;\"> option in the transformers search matcher, which allows for entities to be filtered to only the entity labels specified. This could be useful if only wanting to match against names, or some combination of labels without matching on all of them. Entity labels can vary between different NER models, so it is important to be familiar with the model you are working with.<\/span><\/p>\n<h6><b>Turkish<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">The next example searches text in Turkish for named entities using a different NER model specifically trained for Turkish. The original text is shown below. It contains one name (Ece):<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15569 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-turkish-text-1024x113.png\" alt=\"\" width=\"897\" height=\"99\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-turkish-text-1024x113.png 1024w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-turkish-text-300x33.png 300w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-turkish-text-768x84.png 768w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-turkish-text.png 1600w\" sizes=\"(max-width: 897px) 100vw, 897px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">After sending to the DarkShield API, the named entity <\/span><i><span style=\"font-weight: 400;\">Ece<\/span><\/i><span style=\"font-weight: 400;\"> has been encrypted as \u2018Lse\u2019.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15568 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-masked-turkish-text-1024x105.png\" alt=\"\" width=\"897\" height=\"92\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-masked-turkish-text-1024x105.png 1024w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-masked-turkish-text-300x31.png 300w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-masked-turkish-text-768x79.png 768w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-masked-turkish-text.png 1575w\" sizes=\"(max-width: 897px) 100vw, 897px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-15570 \" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-snip-turkish-annotation-e1643657730449.png\" alt=\"\" width=\"449\" height=\"578\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-snip-turkish-annotation-e1643657730449.png 516w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-snip-turkish-annotation-e1643657730449-233x300.png 233w\" sizes=\"(max-width: 449px) 100vw, 449px\" \/><\/p>\n<h6><b>Japanese<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">My final example demonstrates the use of a Japanese NER model, also available from the Hugging Face model hub, to find a Japanese name in a sentence.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Shown below is the Japanese text I am using for this example, translated into English:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15571 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-japanese-english-translation-text-1024x190.png\" alt=\"\" width=\"749\" height=\"139\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-japanese-english-translation-text-1024x190.png 1024w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-japanese-english-translation-text-300x56.png 300w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-japanese-english-translation-text-768x142.png 768w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-japanese-english-translation-text.png 1274w\" sizes=\"(max-width: 749px) 100vw, 749px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">I set up a search context to utilize the model <\/span><i><span style=\"font-weight: 400;\">KoichiYasuoka\/roberta-base-japanese-luw-upos <\/span><\/i><span style=\"font-weight: 400;\">from the Hugging Face model hub. This model classifies words based on <\/span><a href=\"https:\/\/universaldependencies.org\/u\/pos\/\"><span style=\"font-weight: 400;\">universal part-of-speech<\/span><\/a><span style=\"font-weight: 400;\">, which includes tags such as <\/span><i><span style=\"font-weight: 400;\">VERB<\/span><\/i><span style=\"font-weight: 400;\"> for a verb, <\/span><i><span style=\"font-weight: 400;\">PROPN<\/span><\/i><span style=\"font-weight: 400;\"> for a proper noun, and <\/span><i><span style=\"font-weight: 400;\">ADV<\/span><\/i><span style=\"font-weight: 400;\"> for an adverb.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Using the <\/span><i><span style=\"font-weight: 400;\">entityLabels<\/span><\/i><span style=\"font-weight: 400;\"> property of the transformers search matcher, I am filtering tokens to only proper nouns, which are named entities. I have set the aggregation strategy to simple, as that seems to work best for the grouping of entities in this Japanese NER model.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The masking rule I have set up will completely redact all proper nouns with asterisks:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15572 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-search-context-japanese-ner-1024x299.png\" alt=\"\" width=\"651\" height=\"190\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-search-context-japanese-ner-1024x299.png 1024w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-search-context-japanese-ner-300x88.png 300w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-search-context-japanese-ner-768x224.png 768w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-search-context-japanese-ner.png 1054w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">I had downloaded this model beforehand by using the <\/span><i><span style=\"font-weight: 400;\">model_util.py<\/span><\/i><span style=\"font-weight: 400;\"> script available in the <\/span><i><span style=\"font-weight: 400;\">utils <\/span><\/i><span style=\"font-weight: 400;\">directory of the DarkShield API distribution. Just specify the name of the model using the <\/span><i><span style=\"font-weight: 400;\">-m <\/span><\/i><span style=\"font-weight: 400;\">command line flag, and the name of the tokenizer (the same in this case, and most cases) with the <\/span><i><span style=\"font-weight: 400;\">-t <\/span><\/i><span style=\"font-weight: 400;\">flag.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Once downloaded, either the name of the model in the Hugging Face model hub can be specified in the search matcher, or a path to a folder that contains the model can be specified. If using models from outside the Hugging Face model hub, then specifying a path to a folder that contains all the files associated with the model is the only option.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Here are the results of sending the text shown earlier (Aiko went to the store to buy a banana) to the DarkShield API with the aforementioned contexts.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15573 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-redact-japanese-text.png\" alt=\"\" width=\"331\" height=\"65\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-redact-japanese-text.png 331w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-redact-japanese-text-300x59.png 300w\" sizes=\"(max-width: 331px) 100vw, 331px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Looking at the mask results also returned as a part of the response from the API, Aiko was identified correctly as a name and redacted:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15577 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png\" alt=\"\" width=\"452\" height=\"589\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png 513w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text-230x300.png 230w\" sizes=\"(max-width: 452px) 100vw, 452px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">That concludes an overview and brief demonstration of the newly added transformers search matcher in the DarkShield API as of version 1.4.0.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This article demonstrated just a fraction of the capabilities now possible with the transformers search matcher in the DarkShield API. This new matcher type further expands DarkShield\u2019s NER capabilities by supporting the popular Tensorflow and PyTorch machine learning frameworks.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Contact <\/span><a href=\"mailto:darkshield@iri.com\"><span style=\"font-weight: 400;\">darkshield@iri.com<\/span><\/a><span style=\"font-weight: 400;\"> with any questions or comments about DarkShield and finding named entities with NER models.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Named entity recognition (NER) is a type of machine learning (ML) used to detect named entities within the grammatical context of unstructured text (documents). NER is needed to find things like people names and street addresses, since those do not conform to patterns, nor likely have a match to values in a defined list<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\" title=\"Using Tensorflow and PyTorch NER Models in DarkShield\">Read More<\/a><\/div>\n","protected":false},"author":119,"featured_media":15577,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[8],"tags":[1494,14,1388,1657,1656,1655,1660,1659,1658],"class_list":["post-15546","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-protection","tag-darkshield-api","tag-data-masking","tag-iri-darkshield","tag-japanese","tag-named-entity-recognition","tag-ner","tag-pytorch","tag-tensorflow","tag-turkish"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v23.4 (Yoast SEO v23.4) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Using Tensorflow and PyTorch NER Models in DarkShield - IRI<\/title>\n<meta name=\"description\" content=\"Explore how Tensorflow enhances named entity recognition technology in machine learning for better data protection.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Using Tensorflow and PyTorch NER Models in DarkShield\" \/>\n<meta property=\"og:description\" content=\"Explore how Tensorflow enhances named entity recognition technology in machine learning for better data protection.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2022-01-31T20:07:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-08-12T15:49:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png\" \/>\n\t<meta property=\"og:image:width\" content=\"513\" \/>\n\t<meta property=\"og:image:height\" content=\"668\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Devon Kozenieski\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Devon Kozenieski\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\"},\"author\":{\"name\":\"Devon Kozenieski\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/de972c035aaeecfc40a3ae2ea5ff7ba1\"},\"headline\":\"Using Tensorflow and PyTorch NER Models in DarkShield\",\"datePublished\":\"2022-01-31T20:07:39+00:00\",\"dateModified\":\"2025-08-12T15:49:09+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\"},\"wordCount\":1263,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png\",\"keywords\":[\"Darkshield API\",\"data masking\",\"IRI DarkShield\",\"Japanese\",\"Named entity recognition\",\"NER\",\"Pytorch\",\"TensorFlow\",\"Turkish\"],\"articleSection\":[\"Data Masking\/Protection\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\",\"url\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\",\"name\":\"Using Tensorflow and PyTorch NER Models in DarkShield - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png\",\"datePublished\":\"2022-01-31T20:07:39+00:00\",\"dateModified\":\"2025-08-12T15:49:09+00:00\",\"description\":\"Explore how Tensorflow enhances named entity recognition technology in machine learning for better data protection.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png\",\"width\":513,\"height\":668},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Using Tensorflow and PyTorch NER Models in DarkShield\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.iri.com\/blog\/#website\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/de972c035aaeecfc40a3ae2ea5ff7ba1\",\"name\":\"Devon Kozenieski\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e4c421588c1a85dd9a76146fe15528f7?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e4c421588c1a85dd9a76146fe15528f7?s=96&d=blank&r=g\",\"caption\":\"Devon Kozenieski\"},\"url\":\"https:\/\/www.iri.com\/blog\/author\/devonk\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Using Tensorflow and PyTorch NER Models in DarkShield - IRI","description":"Explore how Tensorflow enhances named entity recognition technology in machine learning for better data protection.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/","og_locale":"en_US","og_type":"article","og_title":"Using Tensorflow and PyTorch NER Models in DarkShield","og_description":"Explore how Tensorflow enhances named entity recognition technology in machine learning for better data protection.","og_url":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/","og_site_name":"IRI","article_published_time":"2022-01-31T20:07:39+00:00","article_modified_time":"2025-08-12T15:49:09+00:00","og_image":[{"width":513,"height":668,"url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png","type":"image\/png"}],"author":"Devon Kozenieski","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Devon Kozenieski","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#article","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/"},"author":{"name":"Devon Kozenieski","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/de972c035aaeecfc40a3ae2ea5ff7ba1"},"headline":"Using Tensorflow and PyTorch NER Models in DarkShield","datePublished":"2022-01-31T20:07:39+00:00","dateModified":"2025-08-12T15:49:09+00:00","mainEntityOfPage":{"@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/"},"wordCount":1263,"commentCount":0,"publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png","keywords":["Darkshield API","data masking","IRI DarkShield","Japanese","Named entity recognition","NER","Pytorch","TensorFlow","Turkish"],"articleSection":["Data Masking\/Protection"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/","url":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/","name":"Using Tensorflow and PyTorch NER Models in DarkShield - IRI","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage"},"image":{"@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png","datePublished":"2022-01-31T20:07:39+00:00","dateModified":"2025-08-12T15:49:09+00:00","description":"Explore how Tensorflow enhances named entity recognition technology in machine learning for better data protection.","breadcrumb":{"@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#primaryimage","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png","width":513,"height":668},{"@type":"BreadcrumbList","@id":"https:\/\/www.iri.com\/blog\/data-protection\/tensorflow-pytorch-ner-models-darkshield\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Using Tensorflow and PyTorch NER Models in DarkShield"}]},{"@type":"WebSite","@id":"https:\/\/www.iri.com\/blog\/#website","url":"https:\/\/www.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.iri.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/www.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/de972c035aaeecfc40a3ae2ea5ff7ba1","name":"Devon Kozenieski","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e4c421588c1a85dd9a76146fe15528f7?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e4c421588c1a85dd9a76146fe15528f7?s=96&d=blank&r=g","caption":"Devon Kozenieski"},"url":"https:\/\/www.iri.com\/blog\/author\/devonk\/"}]}},"jetpack_featured_media_url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2022\/01\/tensorflow-pytorch-darkshield-results-masked-japanese-text.png","_links":{"self":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15546"}],"collection":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/users\/119"}],"replies":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=15546"}],"version-history":[{"count":25,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15546\/revisions"}],"predecessor-version":[{"id":18536,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15546\/revisions\/18536"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media\/15577"}],"wp:attachment":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=15546"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=15546"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=15546"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}