Most-powerful-NLP-library

Name: Most-powerful-NLP-library
Author: FareedKhan-dev

37stars9forks

How to install Most-powerful-NLP-library?

fareedkhan-dev/most-powerful-nlp-library

$git clone https://github.com/FareedKhan-dev/Most-powerful-NLP-library

Installs into the current project.

Summary

Consider one of the most stressful scenarios that most coders face, dealing with large text data that requires cleaning. When using regex, you need to define different sets of patterns to remove text, and even then, you may not be sure if there is any new garbage data that needs

Files · 1

View on GitHub

README.md

1	![Header Image)](https://i.ibb.co/thJLdDt/Group-76.png)
2
3
4	Consider one of the most stressful scenarios that most coders face, dealing with large text data that requires cleaning. When using regex, you need to define different sets of patterns to remove text, and even then, you may not be sure if there is any new garbage data that needs removal. Tasks like these can be stressful for developers because of the time and effort they have to invest, and there’s still uncertainty about whether the new data requires the same procedural coding or not.
5
6	The recent trends of Large Language Models (LLMs), whether open source or closed source, have given us a new dimension of how text data can be handled. Since LLMs can analyze text data more quickly than us and can intelligently understand data to a considerable extent, similar to the way we understand it, why don’t we perform our NLP tasks using LLMs? This could automate the process and make the coder’s life less stressful.
7
8	## The Core Concept Driving My Library
9
10	I tried out several open-source LLMs like [Mistral 8x7b](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) or [LLAMA-2–70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf), but they never met my expectations. Although good for question answering and text generation, they fall short when it comes to NLP tasks. On the other hand, [ChatGPT](https://openai.com/blog/chatgpt) exceeded my expectations, but it requires a paid API access to perform NLP tasks on our custom dataset. [Gemini](https://deepmind.google/technologies/gemini/), as capable as GPT-4, provides a free API with limited access. I tested it with the help of prompt engineering and found that it can solve almost any NLP task you want to tackle.
11
12	Here is a simple visual illustration of how I have used Gemini multi-model to perform NLP tasks on my dataset:
13
14	<img src="https://miro.medium.com/v2/resize:fit:1400/format:webp/1*BmW2FzclMY-pghvFuMTNDg.png" alt="Visual illustration of how my library works">
15
16
17	## list of features avaliable in my library
18
19	for task related to preprocessing
20
21	\| Preprocessing Function \| Description \|
22	\|-------------------------\|----------------------------------------------------------------------------------------------------------\|
23	\| tokenize_text \| Given an input sentence, perform tokenization and return a list of tokens. \|
24	\| lemmatize_text \| Given an input sentence, perform Lemmatization and return the lemmatized sentence. \|
25	\| stem_text \| Given an input sentence, perform Stemming and return the stemmed sentence. \|
26	\| extract_patterns \| Given an input sentence, extract specified patterns and return a list of extracted patterns. \|
27	\| clean_text \| Given an input sentence, clean the text without changing words. \|
28	\| remove_html_tags \| Given an input sentence, remove specified HTML tags. \|
29	\| replace_text \| Given an input sentence, replace specified words using replacement rules. \|
30	\| extract_embeddings \| Given a list of input texts, generate embeddings using a specified model. \|
31
32	for task not related to preprocessing
33
34	\| Core NLP Function \| Description \|
35	\|-------------------------\|----------------------------------------------------------------------------------------------------------\|
36	\| analyze_sentiment \| Perform sentiment analysis on the input text and return the sentiment category. \|
37	\| classify_topic \| Perform Topic Classification on the input text and return the top

People who liked this also viewed

6 picks

Type

Most-powerful-NLP-library

37stars9forks

View on GitHub

How to install Most-powerful-NLP-library?

fareedkhan-dev/most-powerful-nlp-library

$git clone https://github.com/FareedKhan-dev/Most-powerful-NLP-library

Installs into the current project.

Summary

Files · 1

View on GitHub

README.md

1	![Header Image)](https://i.ibb.co/thJLdDt/Group-76.png)
2
3
4	Consider one of the most stressful scenarios that most coders face, dealing with large text data that requires cleaning. When using regex, you need to define different sets of patterns to remove text, and even then, you may not be sure if there is any new garbage data that needs removal. Tasks like these can be stressful for developers because of the time and effort they have to invest, and there’s still uncertainty about whether the new data requires the same procedural coding or not.
5
6	The recent trends of Large Language Models (LLMs), whether open source or closed source, have given us a new dimension of how text data can be handled. Since LLMs can analyze text data more quickly than us and can intelligently understand data to a considerable extent, similar to the way we understand it, why don’t we perform our NLP tasks using LLMs? This could automate the process and make the coder’s life less stressful.
7
8	## The Core Concept Driving My Library
9
10	I tried out several open-source LLMs like [Mistral 8x7b](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) or [LLAMA-2–70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf), but they never met my expectations. Although good for question answering and text generation, they fall short when it comes to NLP tasks. On the other hand, [ChatGPT](https://openai.com/blog/chatgpt) exceeded my expectations, but it requires a paid API access to perform NLP tasks on our custom dataset. [Gemini](https://deepmind.google/technologies/gemini/), as capable as GPT-4, provides a free API with limited access. I tested it with the help of prompt engineering and found that it can solve almost any NLP task you want to tackle.
11
12	Here is a simple visual illustration of how I have used Gemini multi-model to perform NLP tasks on my dataset:
13
14	<img src="https://miro.medium.com/v2/resize:fit:1400/format:webp/1*BmW2FzclMY-pghvFuMTNDg.png" alt="Visual illustration of how my library works">
15
16
17	## list of features avaliable in my library
18
19	for task related to preprocessing
20
21	\| Preprocessing Function \| Description \|
22	\|-------------------------\|----------------------------------------------------------------------------------------------------------\|
23	\| tokenize_text \| Given an input sentence, perform tokenization and return a list of tokens. \|
24	\| lemmatize_text \| Given an input sentence, perform Lemmatization and return the lemmatized sentence. \|
25	\| stem_text \| Given an input sentence, perform Stemming and return the stemmed sentence. \|
26	\| extract_patterns \| Given an input sentence, extract specified patterns and return a list of extracted patterns. \|
27	\| clean_text \| Given an input sentence, clean the text without changing words. \|
28	\| remove_html_tags \| Given an input sentence, remove specified HTML tags. \|
29	\| replace_text \| Given an input sentence, replace specified words using replacement rules. \|
30	\| extract_embeddings \| Given a list of input texts, generate embeddings using a specified model. \|
31
32	for task not related to preprocessing
33
34	\| Core NLP Function \| Description \|
35	\|-------------------------\|----------------------------------------------------------------------------------------------------------\|
36	\| analyze_sentiment \| Perform sentiment analysis on the input text and return the sentiment category. \|
37	\| classify_topic \| Perform Topic Classification on the input text and return the top

People who liked this also viewed

6 picks

Type