Skip to content

[Vertex AI] DecodingError when Citation.uri is not included in response #13518

Closed
@rfarrow1010

Description

@rfarrow1010

Description

I wanted to pass an image along with a text prompt into VertexAI.vertexAI().generativeModel().generateContent() to generate a response. The call itself actually appeared to work fine; I have a log with the response formatted the way I wanted, along with citation metadata. But it appears the citations are causing issues for the SDK. In the logs below, see the first log with the successful response and the citations that do not all follow the same schema. Two of the citations have "startIndex", "endIndex", and "uri", but one of them has no "uri" and instead has a field called "title".
Judging from the second log, it looks like the SDK expects "uri" here, doesn't find it, and crashes before I can retrieve my response. Maybe the decoding can be made more flexible to prevent this?

Reproducing the issue

 let image = UIImage(data: <some photo data>) let config = GenerationConfig(temperature: 0.2) let model = vertex.generativeModel(modelName: "gemini-1.5-flash", generationConfig: config) // to set up the schema: https://firebase.google.com/docs/vertex-ai/function-calling?platform=ios let prompt = """ Return a JSON object that uses the below schema to transcribe a recipe from the given photo. Do not format the JSON with backticks; simply write the JSON object itself in plain text. If there is information you are not sure about, make your best guess. When possible, transcribe ingredients and recipe steps exactly. { title: String creationDate: String (YYYY-MM-DD) description: String ingredients: [String] recipeSteps: [String] estimatedActiveTime: String (HH:MM:SS) estimatedTotalTime: String (HH:MM:SS) minimumServings: Int maximumServings: Int author: String } """ let response = try await model.generateContent(image, prompt) 

Firebase SDK Version

10.26.0

Xcode Version

15.4

Installation Method

Swift Package Manager

Firebase Product(s)

VertexAI

Targeted Platforms

iOS

Relevant Log Output

[FirebaseVertexAI] JSON response: { "candidates": [ { "content": { "role": "model", "parts": [ { "text": "{\n \"title\": \"DIY Grain Salad\",\n \"creationDate\": null,\n \"description\": \"Every Sunday Elyse throws together a hearty grain salad for quick work lunches during the week. The base of her go-to protein-packed salads is typically quinoa or farro and kale and then she dresses it up with any assortment of seasonal veggies. She always tops it off with a generous amount of homemade vinaigrette.\",\n \"ingredients\": [\n \"1 cup whole grains, rinsed\",\n \"1 bunch kale, stems removed, finely chopped\",\n \"2 cups seasonal veggies, chopped or grated\",\n \"1/2 cup something salty (Parmesan, feta, olives)\",\n \"1/2 cup chopped roasted nuts or seeds (almonds, walnuts, pumpkin seeds, sunflower seeds)\",\n \"1 recipe Apple Cider Vinaigrette (page 186)\",\n \"olives\",\n \"cheese\"\n ],\n \"recipeSteps\": [\n \"Cook your grain of choice according to the package directions, being careful not to overcook (you want your grains to have enough texture to hold up to the dressing).\",\n \"For quinoa, we recommend using less water than typically recommended (see our foolproof quinoa cooking method on page 90).\",\n \"Set aside in a large salad bowl to cool.\",\n \"Add the kale, veggies, cheese or olives, and nuts or seeds to the grains and toss to combine.\",\n \"Add the dressing and salt and pepper to taste (go generous on the dressing as the grains will sop it up).\"\n ],\n \"estimatedActiveTime\": \"00:30:00\",\n \"estimatedTotalTime\": \"00:30:00\",\n \"minimumServings\": 5,\n \"maximumServings\": 5,\n \"author\": \"Elyse\"\n}" } ] }, "finishReason": "STOP", "safetyRatings": [ { "category": "HARM_CATEGORY_HATE_SPEECH", "probability": "NEGLIGIBLE", "probabilityScore": 0.13028541, "severity": "HARM_SEVERITY_NEGLIGIBLE", "severityScore": 0.15393034 }, { "category": "HARM_CATEGORY_DANGEROUS_CONTENT", "probability": "NEGLIGIBLE", "probabilityScore": 0.12940271, "severity": "HARM_SEVERITY_NEGLIGIBLE", "severityScore": 0.14057204 }, { "category": "HARM_CATEGORY_HARASSMENT", "probability": "NEGLIGIBLE", "probabilityScore": 0.1334152, "severity": "HARM_SEVERITY_NEGLIGIBLE", "severityScore": 0.091544405 }, { "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "probability": "NEGLIGIBLE", "probabilityScore": 0.1041124, "severity": "HARM_SEVERITY_NEGLIGIBLE", "severityScore": 0.13050689 } ], "citationMetadata": { "citations": [ { "startIndex": 80, "endIndex": 392, "uri": "https://www.pinterest.com/pin/grain-salad--57843176451536516/" }, { "startIndex": 854, "endIndex": 1202, "title": "Your prompt" }, { "startIndex": 1218, "endIndex": 1424, "uri": "https://runfasteatslow.com/blogs/news/fall-inspired-diy-grain-salads" } ] } } ], "usageMetadata": { "promptTokenCount": 411, "candidatesTokenCount": 405, "totalTokenCount": 816 } } Type: Error | Timestamp: 2024-08-12 00:15:27.319474-07:00 | Library: Sous | Subsystem: com.google.firebase.vertex-ai | Category: NetworkResponse | TID: 0x35cc63 [FirebaseVertexAI] Error decoding server JSON: Swift.DecodingError.keyNotFound(CodingKeys(stringValue: "uri", intValue: nil), Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "candidates", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "citationMetadata", intValue: nil), CodingKeys(stringValue: "citations", intValue: nil), _JSONKey(stringValue: "Index 1", intValue: 1)], debugDescription: "No value associated with key CodingKeys(stringValue: \"uri\", intValue: nil) (\"uri\").", underlyingError: nil)) Type: Error | Timestamp: 2024-08-12 00:15:27.322864-07:00 | Library: Sous | Subsystem: com.google.firebase.vertex-ai | Category: | TID: 0x35cc63

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet
{ "originHash" : "aceced254f4af24930664d1a83076c5bafc9df681eccc6d9197ef56542e5907c", "pins" : [ { "identity" : "abseil-cpp-binary", "kind" : "remoteSourceControl", "location" : "https://github.com/google/abseil-cpp-binary.git", "state" : { "revision" : "748c7837511d0e6a507737353af268484e1745e2", "version" : "1.2024011601.1" } }, { "identity" : "app-check", "kind" : "remoteSourceControl", "location" : "https://github.com/google/app-check.git", "state" : { "revision" : "076b241a625e25eac22f8849be256dfb960fcdfe", "version" : "10.19.1" } }, { "identity" : "firebase-ios-sdk", "kind" : "remoteSourceControl", "location" : "https://github.com/firebase/firebase-ios-sdk", "state" : { "revision" : "9d17b500cd98d9a7009751ad62f802e152e97021", "version" : "10.26.0" } }, { "identity" : "fuse-swift", "kind" : "remoteSourceControl", "location" : "https://github.com/krisk/fuse-swift.git", "state" : { "revision" : "26ba868691b2d8b7bf2b1322951eb591be70ccca", "version" : "1.4.0" } }, { "identity" : "googleappmeasurement", "kind" : "remoteSourceControl", "location" : "https://github.com/google/GoogleAppMeasurement.git", "state" : { "revision" : "16244d177c4e989f87b25e9db1012b382cfedc55", "version" : "10.25.0" } }, { "identity" : "googledatatransport", "kind" : "remoteSourceControl", "location" : "https://github.com/google/GoogleDataTransport.git", "state" : { "revision" : "a637d318ae7ae246b02d7305121275bc75ed5565", "version" : "9.4.0" } }, { "identity" : "googleutilities", "kind" : "remoteSourceControl", "location" : "https://github.com/google/GoogleUtilities.git", "state" : { "revision" : "57a1d307f42df690fdef2637f3e5b776da02aad6", "version" : "7.13.3" } }, { "identity" : "grpc-binary", "kind" : "remoteSourceControl", "location" : "https://github.com/google/grpc-binary.git", "state" : { "revision" : "e9fad491d0673bdda7063a0341fb6b47a30c5359", "version" : "1.62.2" } }, { "identity" : "gtm-session-fetcher", "kind" : "remoteSourceControl", "location" : "https://github.com/google/gtm-session-fetcher.git", "state" : { "revision" : "0382ca27f22fb3494cf657d8dc356dc282cd1193", "version" : "3.4.1" } }, { "identity" : "interop-ios-for-google-sdks", "kind" : "remoteSourceControl", "location" : "https://github.com/google/interop-ios-for-google-sdks.git", "state" : { "revision" : "2d12673670417654f08f5f90fdd62926dc3a2648", "version" : "100.0.0" } }, { "identity" : "leveldb", "kind" : "remoteSourceControl", "location" : "https://github.com/firebase/leveldb.git", "state" : { "revision" : "a0bc79961d7be727d258d33d5a6b2f1023270ba1", "version" : "1.22.5" } }, { "identity" : "nanopb", "kind" : "remoteSourceControl", "location" : "https://github.com/firebase/nanopb.git", "state" : { "revision" : "b7e1104502eca3a213b46303391ca4d3bc8ddec1", "version" : "2.30910.0" } }, { "identity" : "promises", "kind" : "remoteSourceControl", "location" : "https://github.com/google/promises.git", "state" : { "revision" : "540318ecedd63d883069ae7f1ed811a2df00b6ac", "version" : "2.4.0" } }, { "identity" : "swift-protobuf", "kind" : "remoteSourceControl", "location" : "https://github.com/apple/swift-protobuf.git", "state" : { "revision" : "9f0c76544701845ad98716f3f6a774a892152bcb", "version" : "1.26.0" } }, { "identity" : "swiftsoup", "kind" : "remoteSourceControl", "location" : "https://github.com/scinfu/SwiftSoup.git", "state" : { "revision" : "e2d11208519549c2e5798d70190472045633f22f", "version" : "2.7.3" } }, { "identity" : "tagcloud", "kind" : "remoteSourceControl", "location" : "https://github.com/yarspirin/TagCloud.git", "state" : { "revision" : "81c1019541c2748cfecff4eabc05ca0bdcaa3713", "version" : "1.0.4" } } ], "version" : 3 } 

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet
Replace this line with the contents of your Podfile.lock!

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    close