Editing article

Title

Summary

Content

<figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*KUZsLKGRzS9-M2dH8GbPbw.png" /></figure>Analyzing customer complaints is crucial for businesses as it enhances customer experience and fosters trust by providing insights into areas that need improvement.Summarization adds value by condensing vast amounts of feedback into actionable insights, enabling businesses to quickly identify trends, prioritize issues, and implement targeted solutions. This efficient process empowers businesses to proactively address customer concerns, improve products or services, and ultimately, improve customer satisfaction and gain more loyalty.In this post, my main goal is to condense lengthy customer complaints (<a href="https://www.consumerfinance.gov/">Consumer Finance Protection Bureau </a>(CFPB) data) and extract relevant important information from them efficiently. I guide you through my utilization of Vertex AI PaLM2 along with LangChain and compare the results of the summarized complaint with an open source LLM (LaMini-Flan-T5–248M) alongside LangChain.<pre>#install required libraries !pip install huggingface-hub !pip install langchain !pip install transformers</pre><pre>#load require librraies import pandas as pd #import hface pipeline from langchain and summarize chain from langchain.llms import HuggingFacePipeline from langchain.chains.summarize import load_summarize_chain from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain import PromptTemplate, LLMChain # load Vertex AI from langchain.llms import VertexAI </pre>Define LLM ModelLLM Model: Vertext AI PaLM 2<a href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text">PaLM 2</a> is Google’s LLM approach to responsible Generative AI and is fine-tuned for different NLP tasks such as classification, summarization, and entity extraction.<pre>#Define Vertex AI PaLM 2 llm to generate response llm = VertexAI(model_name=&#39;text-bison@001&#39;, batch_size=100, #set this if you are using batch processing model_kwargs={&quot;temperature&quot;:0, &quot;max_length&quot;:512} )</pre>LLM Model: LaMini-Flan-T5LaMini-Flan-T5–248 is an open source LLM; a refined iteration of google/flan-t5-base trained on the LaMini-instruction dataset with 2.58M samples.<pre>#defining the lamini model chackpoint in langchain checkpoint = &#39;MBZUAI/LaMini-Flan-T5-248M&#39; #huggingfacepipeline details # Define llm to generate response llm = HuggingFacePipeline.from_model_id(model_id=checkpoint, batch_size=100 #set this if you are using batch processing task =&#39;text2text-generation&#39; model_kwargs={&quot;temperature&quot;:0, &quot;max_length&quot;:512}) </pre>Define Text Splitter and Summarizer ChainSince some of the complaints have long description (that exceed the maximum allowed token size in LLM models), I use LangChain to split them into separate chunks using a “map_reduce” chain type. This will send each chunk separately to LLM in “Map” process, and then “Reduce” function will integrate all summaries together at the end. This is one way to summarize large documents but requires several calls to the LLM. It however, may impact the accuracy and performance.See bellow how I defined a recursive text splitter and a prompt with <a href="https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/">PromptTemplate</a> to guide the LLM to summarize the text.<pre>#define a recursive text spitter to chucnk the complaints text_splitter = RecursiveCharacterTextSplitter( chunk_size = 1000, #I set a to chunck size of 1000 chunk_overlap = 40, length_function = len, ) #set prompt template prompt_template =&quot;&quot;&quot; summarize the given text by high lighting most important information {text} Summary: &quot;&quot;&quot; #define prompt template prompt = PromptTemplate(template=prompt_template, input_variables=[&quot;text&quot;]) #define chain with a map_reduce type chain = load_summarize_chain(llm, map_prompt=prompt, combine_prompt=prompt, verbose=True,chain_type=&quot;map_reduce&quot;) </pre>Summarization can be done in an “online” (for one complaint at a time) or “batch” for a batch/chunk of complaints.Online Summarization:<pre>#for an online mode, just pass one complaint text texts = text_splitter.create_documents([complaint_text]) summary = llm_chain.run(texts) print(summary)</pre>Here you can see an example of a splitted complaint description:<pre>#example of output splitted texts [ Document( page_content=&#39;I am writing to formally complain about inaccurate and illegal reporting of transactions on my credit report, which I believe violates the Fair Credit Reporting Act ( FCRA ) specifically 15 U.S. Code 1681a.I have carefully reviewed my credit report, and I have identified several inaccuracies in the reporting of late payments and utilization of credit. As per 15 U.S. Code 1681a, The term consumer reporting agency means any person which, for monetary fees, dues, or on a cooperative nonprofit basis, regularly engages in whole or in part in the practice of assembling or evaluating consumer credit information or other information on consumers for the purpose of furnishing consumer reports to third parties, and which uses any means or facility of interstate commerce for the purpose of preparing or furnishing consumer reports. \\n The term consumer means an individual. \\n The term consumer report means any written, oral, or other communication of \\nany information by a consumer reporting&#39; ), Document( page_content=&quot;information by a consumer reporting agency bearing on a consumers credit worthiness, credit standing, credit capacity, character, general reputation, personal characteristics, or mode of living. * ( 2 ) Exclusions \\n ( A ) ( i ) report containing information solely as to transactions or experiences between the consumer and the person making the report ; \\n It is illegal to report inaccurate information that adversely affects a consumer &#39;s creditworthiness. Below are the specific discrepancies that I have identified Account number Account type : Home Equity. Date opened Late payments recognized on of . Account number Account type : Credit card. Date opened Late payments recognized on of Account number Account type : Auto Loan. Date opened Late payments recognized on of Account number Account type : Home Equity. Date opened 104 % credit utilization . Account number Account type : Credit card. Date opened 1 % utilization Account number Account type : Credit Card. Date opened 1 %&quot; ), Document( page_content=&#39;type : Credit Card. Date opened 1 % Utilization.\\n I am formally requesting that you conduct a thorough investigation into these matters, as required by the FCRA. I kindly request that you promptly correct the inaccurate information on my credit report by removing the incorrect late payments and adjusting the reported credit utilization. I understand that under 15 U.S. Code 1681i, you are required to conduct a reasonable investigation within 30 days of receiving a dispute. I urge you to adhere to this statutory requirement and provide me with written notification of the results of your investigation. If the investigation confirms the inaccuracies, I request that you update my credit report accordingly and provide me with a revised copy. Additionally, I would appreciate it if you could provide me with information on the steps taken to prevent such errors in the future. If my concerns are not addressed within the stipulated time frame, I will have no choice but to escalate this matter&#39; ), Document( page_content=&#39;no choice but to escalate this matter to the Consumer Financial Protection Bureau. Please treat this matter with the urgency it deserves.&#39; ) ]</pre>and here a pretty good overall summary of important information in the complaint has been highlighted:<pre>#Example of output summary using LaMini-Flan-T5 The person is expressing a complaint about inaccurate and illegal reporting of transactions on their credit report, which violates the Fair Credit Reporting Act (FCRA) specifically 15 U.S. Code 1681a. The person identified several discrepancies in the reporting of late payments and utilization of credit, and is requesting a thorough investigation into the credit card utilization and the FCRA&#39;s requirement to conduct a reasonable investigation within 30 days of receiving a dispute. They request to update their credit report and provide information on steps to prevent future errors.</pre><pre>#Example of output summary using Vertext AI PaLM 2 LLM The writer is writing to formally complain about inaccurate and illegal reporting of transactions on their credit report. The writer believes that this violates the Fair Credit Reporting Act (FCRA). The writer has carefully reviewed their credit report and has identified several inaccuracies in the reporting of late payments and utilization of credit. The writer is requesting that the consumer reporting agency correct the inaccuracies in their credit report.\n\nIt is illegal to report inaccurate information that adversely affects a consumer&#39;s creditworthiness. The specific discrepancies are:\n\n- Account number Account type: Home Equity. Date opened Late payments recognized on of .\n- Account number Account type: Credit card</pre>Batch Summarization:For batch processing, “batch” execution mode of LangChain should be called:<pre>def split_doc(text_splitter,doc): &quot;&quot;&quot; function to split an input document using Langchain Args: text_splitter: a langchain text splitter doc: string text Output: texts: a dictionary of splitted text &quot;&quot;&quot; texts = text_splitter.create_documents([doc]) return texts def summarize_docs(docs,llm_chain): &quot;&quot;&quot; function to summarize chunked documents Args: llm_chain: a langchain summarize chain docs: chunked documents Output: summaries: list of summarized documents &quot;&quot;&quot; #summarize all chunks in one go summary = llm_chain.batch(docs) summaries=[] #extract summaries for summarized_doc in summary: summaries.append(summarized_doc[&#39;output_text&#39;]) return summaries</pre><pre> #load complaints data #read data from file storage df_complaints=read_data() #set the complaint description column for summarizing desc_col=&#39;Consumer complaint narrative&#39; #chunk all complaints docs=df_complaints[desc_col].apply(lambda doc: split_doc(text_splitter,doc) ) #extract and concatenate summaries to the original data df_complaints[&#39;summarized_narrative&#39;]=summarize_docs(docs,llm_chain)</pre>Evaluation:Human assessment plays a crucial role in evaluating summarization tasks. It is essential to thoroughly review the generated output to ensure it is concise and also maintains the core objectives of the original text.To ensure that summaries are aligned closely with human perception, selected samples of summarized documents can be compared with human interpretations, and their <a href="https://www.freecodecamp.org/news/what-is-rouge-and-how-it-works-for-evaluation-of-summaries-e059fb8ac840/#:~:text=ROUGE%20stands%20for%20Recall%2DOriented,as%20well%20as%20machine%20translations.&amp;text=If%20we%20consider%20just%20the,and%20reference%20summary%20is%206.">ROUGE </a>(Recall-Oriented Understudy for Gisting Evaluation) score can be calculated. ROUGE comprises metrics that help effective evaluation of automatic text summarization and machine translations.Final Note:In this article, I’ve showcased the development of a scalable summarization solution. Both LaMini-Flan-T5 and Vertex AI PaLM 2 API (taking into account the<a href="https://cloud.google.com/vertex-ai/generative-ai/pricing"> associated costs</a>) models along with LangChain exhibited strong performance in extracting important highlights from complaints, showcasing robust capabilities in generating AI-powered summaries.In upcoming posts, I’ll employ BERTopic and LLM to identify the predominant trends in customer complaints and uncover the root causes behind these issues. This analysis aims to provide valuable insights for businesses.<img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b14213a6f288" width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/from-genai-to-insights-from-your-customers-part-1-b14213a6f288">From GenAI to Insights from Your Customers (Part 1)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author

Link

Published date

Image url

Feed url

Guid

Hidden blurb

--- !ruby/object:Feedjira::Parser::RSSEntry
title: From GenAI to Insights from Your Customers (Part 1)
url: https://medium.com/google-cloud/from-genai-to-insights-from-your-customers-part-1-b14213a6f288?source=rss----e52cf94d98af---4
author: Tara Pourhabibi
categories:
- text-summarization
- google-cloud-platform
- large-language-models
- generative-ai
- vertex-ai
published: 2024-04-02 01:05:25.000000000 Z
entry_id: !ruby/object:Feedjira::Parser::GloballyUniqueIdentifier
 is_perma_link: 'false'
 guid: https://medium.com/p/b14213a6f288
carlessian_info:
 news_filer_version: 2
 newspaper: Google Cloud - Medium
 macro_region: Blogs
rss_fields:
- title
- url
- author
- categories
- published
- entry_id
- content
content: '<figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*KUZsLKGRzS9-M2dH8GbPbw.png"
 /></figure>Analyzing customer complaints is crucial for businesses as it enhances
 customer experience and fosters trust by providing insights into areas that need
 improvement.Summarization adds value by condensing vast amounts of feedback
 into actionable insights, enabling businesses to quickly identify trends, prioritize
 issues, and implement targeted solutions. This efficient process empowers businesses
 to proactively address customer concerns, improve products or services, and ultimately,
 improve customer satisfaction and gain more loyalty.In this post, my main
 goal is to condense lengthy customer complaints (<a href="https://www.consumerfinance.gov/">Consumer
 Finance Protection Bureau </a>(CFPB) data) and extract relevant important information
 from them efficiently. I guide you through my utilization of Vertex AI PaLM2 along
 with LangChain and compare the results of the summarized complaint with an open
 source LLM (LaMini-Flan-T5–248M) alongside LangChain.<pre>#install required
 libraries !pip install huggingface-hub !pip install langchain !pip
 install transformers</pre><pre>#load require librraies import pandas as pd #import
 hface pipeline from langchain and summarize chain from langchain.llms import
 HuggingFacePipeline from langchain.chains.summarize import load_summarize_chain from
 langchain.text_splitter import RecursiveCharacterTextSplitter from langchain
 import PromptTemplate, LLMChain # load Vertex AI from langchain.llms import
 VertexAI </pre>Define LLM ModelLLM Model:
 Vertext AI PaLM 2<a href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text">PaLM
 2</a> is Google’s LLM approach to responsible Generative AI and is fine-tuned for
 different NLP tasks such as classification, summarization, and entity extraction.<pre>#Define
 Vertex AI PaLM 2 llm to generate response llm = VertexAI(model_name=&#39;text-bison@001&#39;, batch_size=100,
 #set this if you are using batch processing model_kwargs={&quot;temperature&quot;:0,
 &quot;max_length&quot;:512} )</pre>LLM Model:
 LaMini-Flan-T5LaMini-Flan-T5–248 is an open source LLM; a refined
 iteration of google/flan-t5-base trained on the LaMini-instruction dataset with
 2.58M samples.<pre>#defining the lamini model chackpoint in langchain checkpoint
 = &#39;MBZUAI/LaMini-Flan-T5-248M&#39; #huggingfacepipeline details #
 Define llm to generate response llm = HuggingFacePipeline.from_model_id(model_id=checkpoint, batch_size=100
 #set this if you are using batch processing task
 =&#39;text2text-generation&#39; model_kwargs={&quot;temperature&quot;:0,
 &quot;max_length&quot;:512}) </pre>Define Text Splitter and Summarizer
 ChainSince some of the complaints have long description (that exceed
 the maximum allowed token size in LLM models), I use LangChain to split them into
 separate chunks using a “map_reduce” chain type. This will send each chunk separately
 to LLM in “Map” process, and then “Reduce” function will integrate all summaries
 together at the end. This is one way to summarize large documents but requires several
 calls to the LLM. It however, may impact the accuracy and performance.See
 bellow how I defined a recursive text splitter and a prompt with <a href="https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/">PromptTemplate</a>
 to guide the LLM to summarize the text.<pre>#define a recursive text spitter
 to chucnk the complaints text_splitter = RecursiveCharacterTextSplitter( chunk_size
 = 1000, #I set a to chunck size of 1000 chunk_overlap = 40, length_function
 = len, ) #set prompt template prompt_template =&quot;&quot;&quot; summarize
 the given text by high lighting most important information {text} Summary: &quot;&quot;&quot; #define
 prompt template prompt = PromptTemplate(template=prompt_template, input_variables=[&quot;text&quot;]) #define
 chain with a map_reduce type chain = load_summarize_chain(llm, map_prompt=prompt,
 combine_prompt=prompt, verbose=True,chain_type=&quot;map_reduce&quot;) </pre>Summarization
 can be done in an “online” (for one complaint at a time) or “batch” for a batch/chunk
 of complaints.Online Summarization:<pre>#for an online
 mode, just pass one complaint text texts = text_splitter.create_documents([complaint_text]) summary
 = llm_chain.run(texts) print(summary)</pre>Here you can see an example of
 a splitted complaint description:<pre>#example of output splitted texts [ Document( page_content=&#39;I
 am writing to formally complain about inaccurate and illegal reporting of transactions
 on my credit report, which I believe violates the Fair Credit Reporting Act
 ( FCRA ) specifically 15 U.S. Code 1681a.I have carefully reviewed my credit
 report, and I have identified several inaccuracies in the reporting of late payments
 and utilization of credit. As per 15 U.S. Code 1681a, The term consumer reporting
 agency means any person which, for monetary fees, dues, or on a cooperative
 nonprofit basis, regularly engages in whole or in part in the practice of assembling
 or evaluating consumer credit information or other information on consumers for
 the purpose of furnishing consumer reports to third parties, and which uses
 any means or facility of interstate commerce for the purpose of preparing or
 furnishing consumer reports. \\n The term consumer means an individual. \\n The
 term consumer report means any written, oral, or other communication of \\nany
 information by a consumer reporting&#39; ), Document( page_content=&quot;information
 by a consumer reporting agency bearing on a consumers credit worthiness, credit
 standing, credit capacity, character, general reputation, personal characteristics,
 or mode of living. * ( 2 ) Exclusions \\n ( A ) ( i ) report containing information
 solely as to transactions or experiences between the consumer and the person
 making the report ; \\n It is illegal to report inaccurate information that adversely
 affects a consumer &#39;s creditworthiness. Below are the specific discrepancies
 that I have identified Account number Account type : Home Equity. Date opened
 Late payments recognized on of . Account number Account type : Credit card. Date
 opened Late payments recognized on of Account number Account type : Auto Loan. Date
 opened Late payments recognized on of Account number Account type : Home Equity.
 Date opened 104 % credit utilization . Account number Account type : Credit
 card. Date opened 1 % utilization Account number Account type : Credit Card. Date
 opened 1 %&quot; ), Document( page_content=&#39;type :
 Credit Card. Date opened 1 % Utilization.\\n I am formally requesting that you conduct a
 thorough investigation into these matters, as required by the FCRA. I kindly request
 that you promptly correct the inaccurate information on my credit report by
 removing the incorrect late payments and adjusting the reported credit utilization.
 I understand that under 15 U.S. Code 1681i, you are required to conduct a reasonable
 investigation within 30 days of receiving a dispute. I urge you to adhere to
 this statutory requirement and provide me with written notification of the results
 of your investigation. If the investigation confirms the inaccuracies, I request
 that you update my credit report accordingly and provide me with a revised copy.
 Additionally, I would appreciate it if you could provide me with information
 on the steps taken to prevent such errors in the future. If my concerns are
 not addressed within the stipulated time frame, I will have no choice but to escalate
 this matter&#39; ), Document( page_content=&#39;no choice
 but to escalate this matter to the Consumer Financial Protection Bureau. Please
 treat this matter with the urgency it deserves.&#39; ) ]</pre>and
 here a pretty good overall summary of important information in the complaint has
 been highlighted:<pre>#Example of output summary using LaMini-Flan-T5 The
 person is expressing a complaint about inaccurate and illegal reporting of transactions
 on their credit report, which violates the Fair Credit Reporting Act (FCRA) specifically
 15 U.S. Code 1681a. The person identified several discrepancies in the reporting
 of late payments and utilization of credit, and is requesting a thorough investigation
 into the credit card utilization and the FCRA&#39;s requirement to conduct a reasonable
 investigation within 30 days of receiving a dispute. They request to update
 their credit report and provide information on steps to prevent future errors.</pre><pre>#Example
 of output summary using Vertext AI PaLM 2 LLM The writer is writing to formally
 complain about inaccurate and illegal reporting of transactions on their credit
 report. The writer believes that this violates the Fair Credit Reporting Act
 (FCRA). The writer has carefully reviewed their credit report and has identified
 several inaccuracies in the reporting of late payments and utilization of credit.
 The writer is requesting that the consumer reporting agency correct the inaccuracies
 in their credit report.\n\nIt is illegal to report inaccurate information that adversely
 affects a consumer&#39;s creditworthiness. The specific discrepancies are:\n\n-
 Account number Account type: Home Equity. Date opened Late payments recognized on
 of .\n- Account number Account type: Credit card</pre>Batch Summarization:For
 batch processing, “batch” execution mode of LangChain should be called:<pre>def
 split_doc(text_splitter,doc): &quot;&quot;&quot; function to split
 an input document using Langchain Args: text_splitter: a langchain
 text splitter doc: string text Output: texts: a dictionary
 of splitted text &quot;&quot;&quot; texts = text_splitter.create_documents([doc]) return
 texts def summarize_docs(docs,llm_chain): &quot;&quot;&quot; function
 to summarize chunked documents Args: llm_chain: a langchain summarize
 chain docs: chunked documents Output: summaries: list
 of summarized documents &quot;&quot;&quot; #summarize all chunks in
 one go summary = llm_chain.batch(docs) summaries=[] #extract
 summaries for summarized_doc in summary: summaries.append(summarized_doc[&#39;output_text&#39;]) return
 summaries</pre><pre> #load complaints data #read data from file storage df_complaints=read_data() #set
 the complaint description column for summarizing desc_col=&#39;Consumer complaint
 narrative&#39; #chunk all complaints docs=df_complaints[desc_col].apply(lambda
 doc: split_doc(text_splitter,doc) ) #extract and concatenate summaries to
 the original data df_complaints[&#39;summarized_narrative&#39;]=summarize_docs(docs,llm_chain)</pre>Evaluation:Human
 assessment plays a crucial role in evaluating summarization tasks. It is essential
 to thoroughly review the generated output to ensure it is concise and also maintains
 the core objectives of the original text.To ensure that summaries are aligned
 closely with human perception, selected samples of summarized documents can be compared
 with human interpretations, and their <a href="https://www.freecodecamp.org/news/what-is-rouge-and-how-it-works-for-evaluation-of-summaries-e059fb8ac840/#:~:text=ROUGE%20stands%20for%20Recall%2DOriented,as%20well%20as%20machine%20translations.&amp;text=If%20we%20consider%20just%20the,and%20reference%20summary%20is%206.">ROUGE
 </a>(Recall-Oriented Understudy for Gisting Evaluation) score can be calculated.
 ROUGE comprises metrics that help effective evaluation of automatic text summarization
 and machine translations.Final Note:In this article,
 I’ve showcased the development of a scalable summarization solution. Both LaMini-Flan-T5
 and Vertex AI PaLM 2 API (taking into account the<a href="https://cloud.google.com/vertex-ai/generative-ai/pricing">
 associated costs</a>) models along with LangChain exhibited strong performance in
 extracting important highlights from complaints, showcasing robust capabilities
 in generating AI-powered summaries.In upcoming posts, I’ll employ BERTopic
 and LLM to identify the predominant trends in customer complaints and uncover the
 root causes behind these issues. This analysis aims to provide valuable insights
 for businesses.<img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b14213a6f288"
 width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/from-genai-to-insights-from-your-customers-part-1-b14213a6f288">From
 GenAI to Insights from Your Customers (Part 1)</a> was originally published in <a
 href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where
 people are continuing the conversation by highlighting and responding to this story.'

Language

Active

Ricc internal notes

Ricc source

Show this article Back to articles