Editing article

Title

Summary

Content

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0iPZQvd0ZVDuJjeC.jpg" /></figure>This week <a href="https://github.com/langchain4j">LangChain4j</a>, the LLM orchestration framework for Java developers, released version <a href="https://github.com/langchain4j/langchain4j/releases/tag/0.26.1">0.26.1</a>, which contains my first significant contribution to the open source project: support for the Imagen image generation model.Imagen is a text-to-image diffusion model that was <a href="https://imagen.research.google/">announced</a> last year. And it recently upgraded to <a href="https://deepmind.google/technologies/imagen-2/">Imagen v2</a>, with even higher quality graphics generation. As I was curious to integrate it in some of my generative AI projects, I thought that would be a great first <a href="https://github.com/langchain4j/langchain4j/pull/456">contribution</a> to LangChain4j.<blockquote>Caution: At the time of this writing, image generation is still only for allow-listed accounts.</blockquote><blockquote>Furthermore, to run the snippets covered below, you should have an account on Google Cloud Platform, created a project, configured a billing account, enabled the Vertex AI API, and authenticated with the gcloud SDK and the command: gcloud auth application-default login.</blockquote>Now let’s dive in how to use Imagen v1 and v2 with LangChain4j in Java!<h3>Generate your first images</h3>In the following examples, I’m using the following constants, to point at my project details, the endpoint, the region, etc:<pre>private static final String ENDPOINT = &quot;us-central1-aiplatform.googleapis.com:443&quot;; private static final String LOCATION = &quot;us-central1&quot;; private static final String PROJECT = &quot;YOUR_PROJECT_ID&quot;; private static final String PUBLISHER = &quot;google&quot;;</pre>First, we’re going to create an instance of the model:<pre>VertexAiImageModel imagenModel = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@005&quot;) .maxRetries(2) .withPersisting() .build();</pre>There are 2 models you can use:<ul><li>imagegeneration@005 corresponds to Imagen 2</li><li>imagegeneration@002 is the previous version (Imagen 1)</li></ul>In this article, we’ll use both models. Why? Because currently Imagen 2 doesn’t support image editing, so we’ll have to use Imagen 1 for that purpose.The configuration above uses withPersisting() to save the generated images in a temporary folder on your system. If you don&#39;t persist the image files, the content of the image is avaiable as Base 64 encoded bytes in the Images objects returned. You can also specify persistTo(somePath) to specify a particular directory where you want the generated files to be saved.Let’s create our first image:<pre>Response&lt;Image&gt; imageResponse = imagenModel.generate( &quot;watercolor of a colorful parrot drinking a cup of coffee&quot;);</pre>The Response object wraps the created Image. You can get the Image by calling imageResponse.getContent(). And you can retrieve the URL of the image (if saved locally) with imageResponse.getContent().url(). The Base 64 encoded bytes can be retrieved with imageResponse.getContent().base64Data()Some other tweaks to the model configuration:<ul><li>Specify the language of the prompt: language(&quot;ja&quot;) (if the language is not officially supported, it&#39;s usually translated back to English anyway).</li><li>Define a negative prompt with things you don’t want to see in the picture: negativePrompt(&quot;black feathers&quot;).</li><li>Use a particular seed to always generate the same image with the same seed: seed(1234L).</li></ul>So if you want to generate a picture of a pizza with a prompt in Japanese, but you don’t want to have pepperoni and pineapple, you could configure your model and generate as follows:<pre>VertexAiImageModel imagenModel = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@005&quot;) .language(&quot;ja&quot;) .negativePrompt(&quot;pepperoni, pineapple&quot;) .maxRetries(2) .withPersisting() .build(); Response&lt;Image&gt; imageResponse = imagenModel.generate(&quot;ピザ&quot;); // pizza</pre><h3>Image editing with Imagen 1</h3>With Imagen 1, you can <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/image/edit-images?hl=en">edit</a> existing images:<ul><li>mask-based editing: you can specify a mask, a black &amp; white image where the white parts are the corresponding parts of the original image that should be edited,</li><li>mask free editing: where you just give a prompt and let the model figure out what should be edited on its own or following the prompt.</li></ul>When generating and editing with Imagen 1, you can also configure the model to use a particular style (with Imagen 2, you just specify it in the prompt) with sampleImageStyle(VertexAiImageModel.ImageStyle.photograph):- photograph - digital_art - landscape - sketch - watercolor - cyberpunk - pop_artWhen editing an image, you may wish to decide how strong or not the modification should be, with .guidanceScale(100). Usually, between 0 and 20 or so, it&#39;s lightly edited, between 20 and 100 it&#39;s getting more impactful edits, and 100 and above it&#39;s the maximum edition level.Let’s say I generated an image of a lush forrest (I’ll use that as my original image):<pre>VertexAiImageModel model = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@002&quot;) .seed(19707L) .sampleImageStyle(VertexAiImageModel.ImageStyle.photograph) .guidanceScale(100) .maxRetries(4) .withPersisting() .build(); Response&lt;Image&gt; forestResp = model.generate(&quot;lush forest&quot;);</pre>Now I want to edit my forrest to add a small red tree in the bottom of the image. I’m loading a black and white mask image with a white square at the bottom. And I pass the original image, the mask image, and the modification prompt, to the new edit() method:<pre>URI maskFileUri = getClass().getClassLoader().getResource(&quot;mask.png&quot;).toURI(); Response&lt;Image&gt; compositeResp = model.edit( forestResp.content(), // original image to edit fromPath(Paths.get(maskFileUri)), // the mask image &quot;red trees&quot; // the new prompt );</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AfQ24hvdH9hoTLsT.jpg" /></figure>Another kind of editing you can do is to upscale an existing image. As far as I know, it’s only supported for Imagen v1 for now, so we’ll continue with that model.In this example, we’ll generate an image of 1024x1024 pixels, and we’ll scale it to 4096x4096:<pre>VertexAiImageModel imagenModel = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@002&quot;) .sampleImageSize(1024) .withPersisting() .persistTo(defaultTempDirPath) .maxRetries(3) .build(); Response&lt;Image&gt; imageResponse = imagenModel.generate(&quot;A black bird looking itself in an antique mirror&quot;); VertexAiImageModel imagenModelForUpscaling = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@002&quot;) .sampleImageSize(4096) .withPersisting() .persistTo(defaultTempDirPath) .maxRetries(3) .build(); Response&lt;Image&gt; upscaledImageResponse = imagenModelForUpscaling.edit(imageResponse.content(), &quot;&quot;);</pre>And now you have a much bigger image!<h3>Conclusion</h3>That’s about it for image generation and editing with Imagen in LangChain4j today! Be sure to use LangChain4j v0.26.1 which contains that new integration. And I’m looking forward to seeing the pictures you generate with it!Originally published at <a href="https://glaforge.dev/posts/2024/02/01/image-generation-with-imagen-and-langchain4j/">https://glaforge.dev</a> on February 1, 2024.<img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61ca08ae6aac" width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/image-generation-with-imagen-and-langchain4j-61ca08ae6aac">Image generation with Imagen and LangChain4j</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author

Link

Published date

Image url

Feed url

Guid

Hidden blurb

--- !ruby/object:Feedjira::Parser::RSSEntry
title: Image generation with Imagen and LangChain4j
url: https://medium.com/google-cloud/image-generation-with-imagen-and-langchain4j-61ca08ae6aac?source=rss-431147437aeb------2
author: Guillaume Laforge
categories:
- google-cloud-platform
- imagen
- java
- langchain
- generative-ai-use-cases
published: 2024-02-01 00:00:35.000000000 Z
entry_id: !ruby/object:Feedjira::Parser::GloballyUniqueIdentifier
 is_perma_link: 'false'
 guid: https://medium.com/p/61ca08ae6aac
carlessian_info:
 news_filer_version: 2
 newspaper: Guillaume Laforge - Medium
 macro_region: Blogs
rss_fields:
- title
- url
- author
- categories
- published
- entry_id
- content
content: '<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0iPZQvd0ZVDuJjeC.jpg"
 /></figure>This week <a href="https://github.com/langchain4j">LangChain4j</a>,
 the LLM orchestration framework for Java developers, released version <a href="https://github.com/langchain4j/langchain4j/releases/tag/0.26.1">0.26.1</a>,
 which contains my first significant contribution to the open source project: support
 for the Imagen image generation model.Imagen is
 a text-to-image diffusion model that was <a href="https://imagen.research.google/">announced</a>
 last year. And it recently upgraded to <a href="https://deepmind.google/technologies/imagen-2/">Imagen
 v2</a>, with even higher quality graphics generation. As I was curious to integrate
 it in some of my generative AI projects, I thought that would be a great first <a
 href="https://github.com/langchain4j/langchain4j/pull/456">contribution</a> to LangChain4j.<blockquote>Caution:
 At the time of this writing, image generation is still only for allow-listed accounts.</blockquote><blockquote>Furthermore,
 to run the snippets covered below, you should have an account on Google Cloud Platform,
 created a project, configured a billing account, enabled the Vertex AI API, and
 authenticated with the gcloud SDK and the command: gcloud auth application-default
 login.</blockquote>Now let’s dive in how to use Imagen v1 and v2 with LangChain4j
 in Java!<h3>Generate your first images</h3>In the following examples, I’m
 using the following constants, to point at my project details, the endpoint, the
 region, etc:<pre>private static final String ENDPOINT = &quot;us-central1-aiplatform.googleapis.com:443&quot;; private
 static final String LOCATION = &quot;us-central1&quot;; private static final
 String PROJECT = &quot;YOUR_PROJECT_ID&quot;; private static final String PUBLISHER
 = &quot;google&quot;;</pre>First, we’re going to create an instance of the model:<pre>VertexAiImageModel
 imagenModel = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@005&quot;) .maxRetries(2) .withPersisting() .build();</pre>There
 are 2 models you can use:<ul><li>imagegeneration@005 corresponds to Imagen 2</li><li>imagegeneration@002
 is the previous version (Imagen 1)</li></ul>In this article, we’ll use both models.
 Why? Because currently Imagen 2 doesn’t support image editing, so we’ll have to
 use Imagen 1 for that purpose.The configuration above uses withPersisting()
 to save the generated images in a temporary folder on your system. If you don&#39;t
 persist the image files, the content of the image is avaiable as Base 64 encoded
 bytes in the Images objects returned. You can also specify persistTo(somePath) to
 specify a particular directory where you want the generated files to be saved.Let’s
 create our first image:<pre>Response&lt;Image&gt; imageResponse = imagenModel.generate( &quot;watercolor
 of a colorful parrot drinking a cup of coffee&quot;);</pre>The Response object
 wraps the created Image. You can get the Image by calling imageResponse.getContent().
 And you can retrieve the URL of the image (if saved locally) with imageResponse.getContent().url().
 The Base 64 encoded bytes can be retrieved with imageResponse.getContent().base64Data()Some
 other tweaks to the model configuration:<ul><li>Specify the language
 of the prompt: language(&quot;ja&quot;) (if the language is not officially supported,
 it&#39;s usually translated back to English anyway).</li><li>Define a negative
 prompt with things you don’t want to see in the picture: negativePrompt(&quot;black
 feathers&quot;).</li><li>Use a particular seed to always generate
 the same image with the same seed: seed(1234L).</li></ul>So if you want to generate
 a picture of a pizza with a prompt in Japanese, but you don’t want to have pepperoni
 and pineapple, you could configure your model and generate as follows:<pre>VertexAiImageModel
 imagenModel = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@005&quot;) .language(&quot;ja&quot;) .negativePrompt(&quot;pepperoni,
 pineapple&quot;) .maxRetries(2) .withPersisting() .build(); Response&lt;Image&gt;
 imageResponse = imagenModel.generate(&quot;ピザ&quot;); // pizza</pre><h3>Image editing
 with Imagen 1</h3>With Imagen 1, you can <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/image/edit-images?hl=en">edit</a>
 existing images:<ul><li>mask-based editing: you can specify
 a mask, a black &amp; white image where the white parts are the corresponding parts
 of the original image that should be edited,</li><li>mask free editing:
 where you just give a prompt and let the model figure out what should be edited
 on its own or following the prompt.</li></ul>When generating and editing with
 Imagen 1, you can also configure the model to use a particular style (with Imagen
 2, you just specify it in the prompt) with sampleImageStyle(VertexAiImageModel.ImageStyle.photograph):-
 photograph - digital_art - landscape - sketch - watercolor - cyberpunk - pop_artWhen
 editing an image, you may wish to decide how strong or not the modification should
 be, with .guidanceScale(100). Usually, between 0 and 20 or so, it&#39;s lightly
 edited, between 20 and 100 it&#39;s getting more impactful edits, and 100 and above
 it&#39;s the maximum edition level.Let’s say I generated an image of a lush
 forrest (I’ll use that as my original image):<pre>VertexAiImageModel model =
 VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@002&quot;) .seed(19707L) .sampleImageStyle(VertexAiImageModel.ImageStyle.photograph) .guidanceScale(100) .maxRetries(4) .withPersisting() .build(); Response&lt;Image&gt;
 forestResp = model.generate(&quot;lush forest&quot;);</pre>Now I want to edit
 my forrest to add a small red tree in the bottom of the image. I’m loading a black
 and white mask image with a white square at the bottom. And I pass the original
 image, the mask image, and the modification prompt, to the new edit() method:<pre>URI
 maskFileUri = getClass().getClassLoader().getResource(&quot;mask.png&quot;).toURI(); Response&lt;Image&gt;
 compositeResp = model.edit( forestResp.content(), // original
 image to edit fromPath(Paths.get(maskFileUri)), // the mask image &quot;red
 trees&quot; // the new prompt );</pre><figure><img alt=""
 src="https://cdn-images-1.medium.com/max/1024/0*AfQ24hvdH9hoTLsT.jpg" /></figure>Another
 kind of editing you can do is to upscale an existing image. As far as I know, it’s
 only supported for Imagen v1 for now, so we’ll continue with that model.In
 this example, we’ll generate an image of 1024x1024 pixels, and we’ll scale it to
 4096x4096:<pre>VertexAiImageModel imagenModel = VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@002&quot;) .sampleImageSize(1024) .withPersisting() .persistTo(defaultTempDirPath) .maxRetries(3) .build(); Response&lt;Image&gt;
 imageResponse = imagenModel.generate(&quot;A black bird looking itself
 in an antique mirror&quot;); VertexAiImageModel imagenModelForUpscaling =
 VertexAiImageModel.builder() .endpoint(ENDPOINT) .location(LOCATION) .project(PROJECT) .publisher(PUBLISHER) .modelName(&quot;imagegeneration@002&quot;) .sampleImageSize(4096) .withPersisting() .persistTo(defaultTempDirPath) .maxRetries(3) .build(); Response&lt;Image&gt;
 upscaledImageResponse = imagenModelForUpscaling.edit(imageResponse.content(),
 &quot;&quot;);</pre>And now you have a much bigger image!<h3>Conclusion</h3>That’s
 about it for image generation and editing with Imagen in LangChain4j
 today! Be sure to use LangChain4j v0.26.1 which contains that new
 integration. And I’m looking forward to seeing the pictures you generate with it!Originally
 published at <a href="https://glaforge.dev/posts/2024/02/01/image-generation-with-imagen-and-langchain4j/">https://glaforge.dev</a>
 on February 1, 2024.<img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61ca08ae6aac"
 width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/image-generation-with-imagen-and-langchain4j-61ca08ae6aac">Image
 generation with Imagen and LangChain4j</a> was originally published in <a href="https://medium.com/google-cloud">Google
 Cloud - Community</a> on Medium, where people are continuing the conversation by
 highlighting and responding to this story.'

Language

Active

Ricc internal notes

Ricc source

Show this article Back to articles