[60일차] ASAC 빅데이터 분석가 4기(DL-9)

⚙️ Transformers - NLP

📌 기본 세팅

!git clone https://github.com/rickiepark/nlp-with-transformers.git
%cd nlp-with-transformers
from install import *
install_requirements(chapter=1)

text = """Dear Amazon, last week I ordered an Optimus Prime action figure \
from your online store in Germany. Unfortunately, when I opened the package, \
I discovered to my horror that I had been sent an action figure of Megatron \
instead! As a lifelong enemy of the Decepticons, I hope you can understand my \
dilemma. To resolve the issue, I demand an exchange of Megatron for the \
Optimus Prime figure I ordered. Enclosed are copies of my records concerning \
this purchase. I expect to hear from you soon. Sincerely, Bumblebee."""

access_token="허깅페이스 access token"

---

📌 텍스트 분류

from transformers import pipeline
classifier = pipeline("text-classification", token=access_token)

import pandas as pd
outputs = classifier(text)
pd.DataFrame(outputs)

텍스트 분류 결과

---

📌 텍스트 객체 추출

ner_tagger = pipeline("ner", aggregation_strategy="simple", token=access_token)

outputs = ner_tagger(text)
print(text)
print("-"*80)
pd.DataFrame(outputs)

텍스트 객체 추출 결과

---

📌 텍스트 기반 Q&A

reader = pipeline("question-answering", token=access_token)

question = "What does the customer want?"
outputs = reader(question=question, context=text)
pd.DataFrame([outputs])

Q&A 결과

question = "Where did he buy the Megatron?"
outputs = reader(question=question, context=text)
pd.DataFrame([outputs])

Q&A 결과

---

📌 텍스트 요약

summarizer = pipeline("summarization", token=access_token)

outputs = summarizer(text, max_length=56, clean_up_tokenization_spaces=True)
print(outputs[0]['summary_text'])

텍스트 요약 결과

---

📌 텍스트 번역

translator = pipeline("translation_en_to_de",
                      model="Helsinki-NLP/opus-mt-en-de", token=access_token)

outputs = translator(text, clean_up_tokenization_spaces=True, min_length=100)
print(outputs[0]['translation_text'])

텍스트 번역 결과 (영 > 독)

translator2 = pipeline("translation", model="Helsinki-NLP/opus-mt-ko-en", token=access_token)
print(translator2("오늘의 날씨가 참 좋습니다")[0]['translation_text'])

텍스트 번역 결과 (한 > 영)

---

📌 텍스트 생성

generator = pipeline("text-generation", token=access_token)

response = "Dear Bumblebee, I am sorry to hear that your order was mixed up."
prompt = text + "\n\nCustomer service response:\n" + response
outputs = generator(prompt, max_length=200)
print(outputs[0]['generated_text'])

텍스트 생성 결과