![image](https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cloud/notebooks/headers/watsonx-Prompt_Lab-Notebook.png)
# Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service
#### Disclaimers

- Use only Projects and Spaces that are available in watsonx context.


## Notebook content

This notebook provides a detailed demonstration of the steps and code required to showcase support for watsonx.ai AI service.

Some familiarity with Python is helpful. This notebook uses Python 3.11.


## Learning goal

The learning goal for your notebook is to leverage AI services to generate accurate and contextually relevant responses based on a given image and a related question.


## Table of Contents

This notebook contains the following parts:

- [Setup](#setup)
- [Create AI service](#ai_service)
- [Testing AI service's function locally](#testing)
- [Deploy AI service](#deploy)
- [Example of Executing an AI service](#example)
- [Summary](#summary)
<a id="setup"></a>
## Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

-  Create a <a href="https://cloud.ibm.com/catalog/services/watsonxai-runtime" target="_blank" rel="noopener no referrer">watsonx.ai Runtime Service</a> instance (a free plan is offered and information about how to create the instance can be found <a href="https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html?context=wx&audience=wdp" target="_blank" rel="noopener no referrer">here</a>).
### Install and import the `datasets` and dependencies
!pip install -U "ibm_watsonx_ai>=1.1.22" | tail -n 1
### Define the watsonx.ai credentials
Use the code cell below to define the watsonx.ai credentials that are required to work with watsonx Foundation Model inferencing.

**Action:** Provide the IBM Cloud user API key. For details, see <a href="https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui" target="_blank" rel="noopener no referrer">Managing user API keys</a>.
import getpass
from ibm_watsonx_ai import Credentials

credentials = Credentials(
    url="https://us-south.ml.cloud.ibm.com",
    api_key=getpass.getpass("Enter your watsonx.ai api key and hit enter: "),
)
### Working with spaces

You need to create a space that will be used for your work. If you do not have a space, you can use [Deployment Spaces Dashboard](https://dataplatform.cloud.ibm.com/ml-runtime/spaces?context=wx) to create one.

- Click **New Deployment Space**
- Create an empty space
- Select Cloud Object Storage
- Select watsonx.ai Runtime instance and press **Create**
- Go to **Manage** tab
- Copy `Space GUID` and paste it below

**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cloud/notebooks/python_sdk/instance-management/Space%20management.ipynb).

**Action**: assign space ID below
import os

try:
    space_id = os.environ["SPACE_ID"]
except KeyError:
    space_id = input("Please enter your space_id (hit enter): ")
Create an instance of APIClient with authentication details.
from ibm_watsonx_ai import APIClient

api_client = APIClient(credentials=credentials, space_id=space_id)
Specify the `model_id` of the model you will use for the chat with image.
model_id = "meta-llama/llama-3-2-11b-vision-instruct"
<a id="ai_service"></a>
## Create AI service

Prepare function which will be deployed using AI service.

Please specify the default parameters that will be passed to the function.
def deployable_ai_service(context, space_id=space_id, url=credentials["url"], model_id=model_id, params={"temperature": 1}, **kwargs):

    import requests
    import base64
    from ibm_watsonx_ai import APIClient, Credentials
    from ibm_watsonx_ai.foundation_models import ModelInference

    api_client = APIClient(
        credentials=Credentials(url=url, token=context.generate_token()),
        space_id=space_id,
    )

    model = ModelInference(
        model_id=model_id,
        api_client=api_client,
        params=params,
    )

    def generate(context) -> dict:

        api_client.set_token(context.get_token())

        payload = context.get_json()
        question = payload["question"]
        image_url = payload["image_url"]

        response = requests.get(image_url)
        response.raise_for_status()
        base64_image = base64.b64encode(response.content).decode('utf-8')

        messages = [
            {
                "role": "user",
                "content": [
                {
                    "type": "text",
                    "text": question
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64," + base64_image,
                        "detail": "auto"
                    }
                }
                ]
            }
        ]


        response = model.chat(messages=messages)

        return {
            "body": response
            }

    def generate_stream(context):

        api_client.set_token(context.get_token())

        payload = context.get_json()
        question = payload["question"]
        image_url = payload["image_url"]

        response = requests.get(image_url)
        response.raise_for_status()
        base64_image = base64.b64encode(response.content).decode('utf-8')

        messages = [
            {
                "role": "user",
                "content": [
                {
                    "type": "text",
                    "text": question
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64," + base64_image,
                        "detail": "auto"
                    }
                }
                ]
            }
        ]

        yield from model.chat_stream(messages)

    return generate, generate_stream
<a id="testing"></a>
## Testing AI service's function locally

You can test AI service's function locally. Initialise `RuntimeContext` firstly.
from ibm_watsonx_ai.deployments import RuntimeContext

context = RuntimeContext(api_client=api_client)
local_function = deployable_ai_service(context=context)
Please retrieve an image and display it. This example is based on the IBM logo.
import requests
from IPython.display import Image

image_url = "https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/logo/ibm_logo.jpg"

response = requests.get(image_url)

Image(url=image_url, width=600)
Prepare request json payload for local invoke.
context.request_payload_json = {"question": "Describe the image", "image_url": image_url}
Execute the `generate` function locally.
resp = local_function[0](context)
resp
Execute the `generate_stream` function locally.
response = local_function[1](context)
for chunk in response:
    if chunk["choices"]:
        print(chunk["choices"][0]["delta"].get("content", ""), end="", flush=True)
<a id="deploy"></a>
## Deploy AI service
Store AI service with previous created custom software specifications.
sw_spec_id = api_client.software_specifications.get_id_by_name("runtime-24.1-py3.11")
sw_spec_id
meta_props = {
    api_client.repository.AIServiceMetaNames.NAME: "AI service with SDK",
    api_client.repository.AIServiceMetaNames.SOFTWARE_SPEC_ID: sw_spec_id
}
stored_ai_service_details = api_client.repository.store_ai_service(deployable_ai_service, meta_props)
ai_service_id = api_client.repository.get_ai_service_id(stored_ai_service_details)
ai_service_id
Create online deployment of AI service.
meta_props = {
    api_client.deployments.ConfigurationMetaNames.NAME: "AI service with SDK",
    api_client.deployments.ConfigurationMetaNames.ONLINE: {},
}

deployment_details = api_client.deployments.create(ai_service_id, meta_props)
Obtain the `deployment_id` of the previously created deployment.
deployment_id = api_client.deployments.get_id(deployment_details)
<a id="example"></a>
## Example of Executing an AI service.

Execute `generate` method.
question = "Describe the image"

deployments_results = api_client.deployments.run_ai_service(
    deployment_id, {"question": question, "image_url": image_url}
)
import json

print(json.dumps(deployments_results, indent=2))
Execute `generate_stream` method.
question = "Describe the image"

deployments_results = api_client.deployments.run_ai_service_stream(
    deployment_id, {"question": question, "image_url": image_url}
)
import json

for chunk in deployments_results:
    chunk_json = json.loads(chunk)
    if chunk_json["choices"]:
        print(chunk_json["choices"][0]["delta"].get("content", ""), end="", flush=True)
<a id="summary"></a>
## Summary and next steps

You successfully completed this notebook!

You learned how to create and deploy AI service using `ibm_watsonx_ai` SDK.

Check out our _<a href="https://ibm.github.io/watsonx-ai-python-sdk/samples.html" target="_blank" rel="noopener no referrer">Online Documentation</a>_ for more samples, tutorials, documentation, how-tos, and blog posts. 
### Author

**Mateusz Szewczyk**, Software Engineer at watsonx.ai.
Copyright © 2024-2025 IBM. This notebook and its source code are released under the terms of the MIT License.