You can use Firebase ML to recognize well-known landmarks in an image.
Before you begin
- If you haven't already, add Firebase to your Android project.
- In your module (app-level) Gradle file (usually
<project>/<app-module>/build.gradle.ktsor<project>/<app-module>/build.gradle), add the dependency for the Firebase ML Vision library for Android. We recommend using the Firebase Android BoM to control library versioning.dependencies{// Import the BoM for the Firebase platformimplementation(platform("com.google.firebase:firebase-bom:33.12.0"))// Add the dependency for the Firebase ML Vision library// When using the BoM, you don't specify versions in Firebase library dependenciesimplementation'com.google.firebase:firebase-ml-vision'}
By using the Firebase Android BoM, your app will always use compatible versions of Firebase Android libraries.
(Alternative) Add Firebase library dependencies without using the BoM
If you choose not to use the Firebase BoM, you must specify each Firebase library version in its dependency line.
Note that if you use multiple Firebase libraries in your app, we strongly recommend using the BoM to manage library versions, which ensures that all versions are compatible.
dependencies{// Add the dependency for the Firebase ML Vision library// When NOT using the BoM, you must specify versions in Firebase library dependenciesimplementation'com.google.firebase:firebase-ml-vision:24.1.0'}
If you haven't already enabled Cloud-based APIs for your project, do so now:
- Open the Firebase ML APIs page in the Firebase console.
If you haven't already upgraded your project to the pay-as-you-go Blaze pricing plan, click Upgrade to do so. (You'll be prompted to upgrade only if your project isn't on the Blaze pricing plan.)
Only projects on the Blaze pricing plan can use Cloud-based APIs.
- If Cloud-based APIs aren't already enabled, click Enable Cloud-based APIs.
Configure the landmark detector
By default, the Cloud detector uses the STABLE version of the model and returns up to 10 results. If you want to change either of these settings, specify them with a FirebaseVisionCloudDetectorOptions object.
For example, to change both of the default settings, build a FirebaseVisionCloudDetectorOptions object as in the following example:
Kotlin
valoptions=FirebaseVisionCloudDetectorOptions.Builder().setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL).setMaxResults(15).build()
Java
FirebaseVisionCloudDetectorOptionsoptions=newFirebaseVisionCloudDetectorOptions.Builder().setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL).setMaxResults(15).build();
To use the default settings, you can use FirebaseVisionCloudDetectorOptions.DEFAULT in the next step.
Run the landmark detector
To recognize landmarks in an image, create aFirebaseVisionImage object from either a Bitmap, media.Image, ByteBuffer, byte array, or a file on the device. Then, pass the FirebaseVisionImage object to the FirebaseVisionCloudLandmarkDetector's detectInImage method.Create a
FirebaseVisionImageobject from your image.To create a
FirebaseVisionImageobject from amedia.Imageobject, such as when capturing an image from a device's camera, pass themedia.Imageobject and the image's rotation toFirebaseVisionImage.fromMediaImage().If you use the CameraX library, the
OnImageCapturedListenerandImageAnalysis.Analyzerclasses calculate the rotation value for you, so you just need to convert the rotation to one of Firebase ML'sROTATION_constants before callingFirebaseVisionImage.fromMediaImage():Kotlin
privateclassYourImageAnalyzer:ImageAnalysis.Analyzer{privatefundegreesToFirebaseRotation(degrees:Int):Int=when(degrees){0->FirebaseVisionImageMetadata.ROTATION_090->FirebaseVisionImageMetadata.ROTATION_90180->FirebaseVisionImageMetadata.ROTATION_180270->FirebaseVisionImageMetadata.ROTATION_270else->throwException("Rotation must be 0, 90, 180, or 270.")}overridefunanalyze(imageProxy:ImageProxy?,degrees:Int){valmediaImage=imageProxy?.imagevalimageRotation=degreesToFirebaseRotation(degrees)if(mediaImage!=null){valimage=FirebaseVisionImage.fromMediaImage(mediaImage,imageRotation)// Pass image to an ML Vision API// ...}}}
Java
privateclassYourAnalyzerimplementsImageAnalysis.Analyzer{privateintdegreesToFirebaseRotation(intdegrees){switch(degrees){case0:returnFirebaseVisionImageMetadata.ROTATION_0;case90:returnFirebaseVisionImageMetadata.ROTATION_90;case180:returnFirebaseVisionImageMetadata.ROTATION_180;case270:returnFirebaseVisionImageMetadata.ROTATION_270;default:thrownewIllegalArgumentException("Rotation must be 0, 90, 180, or 270.");}}@Overridepublicvoidanalyze(ImageProxyimageProxy,intdegrees){if(imageProxy==null||imageProxy.getImage()==null){return;}ImagemediaImage=imageProxy.getImage();introtation=degreesToFirebaseRotation(degrees);FirebaseVisionImageimage=FirebaseVisionImage.fromMediaImage(mediaImage,rotation);// Pass image to an ML Vision API// ...}}
If you don't use a camera library that gives you the image's rotation, you can calculate it from the device's rotation and the orientation of camera sensor in the device:
Kotlin
privatevalORIENTATIONS=SparseIntArray()init{ORIENTATIONS.append(Surface.ROTATION_0,90)ORIENTATIONS.append(Surface.ROTATION_90,0)ORIENTATIONS.append(Surface.ROTATION_180,270)ORIENTATIONS.append(Surface.ROTATION_270,180)}/** * Get the angle by which an image must be rotated given the device's current * orientation. */@RequiresApi(api=Build.VERSION_CODES.LOLLIPOP)@Throws(CameraAccessException::class)privatefungetRotationCompensation(cameraId:String,activity:Activity,context:Context):Int{// Get the device's current rotation relative to its "native" orientation.// Then, from the ORIENTATIONS table, look up the angle the image must be// rotated to compensate for the device's rotation.valdeviceRotation=activity.windowManager.defaultDisplay.rotationvarrotationCompensation=ORIENTATIONS.get(deviceRotation)// On most devices, the sensor orientation is 90 degrees, but for some// devices it is 270 degrees. For devices with a sensor orientation of// 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.valcameraManager=context.getSystemService(CAMERA_SERVICE)asCameraManagervalsensorOrientation=cameraManager.getCameraCharacteristics(cameraId).get(CameraCharacteristics.SENSOR_ORIENTATION)!!rotationCompensation=(rotationCompensation+sensorOrientation+270)%360// Return the corresponding FirebaseVisionImageMetadata rotation value.valresult:Intwhen(rotationCompensation){0->result=FirebaseVisionImageMetadata.ROTATION_090->result=FirebaseVisionImageMetadata.ROTATION_90180->result=FirebaseVisionImageMetadata.ROTATION_180270->result=FirebaseVisionImageMetadata.ROTATION_270else->{result=FirebaseVisionImageMetadata.ROTATION_0Log.e(TAG,"Bad rotation value: $rotationCompensation")}}returnresult}
Java
privatestaticfinalSparseIntArrayORIENTATIONS=newSparseIntArray();static{ORIENTATIONS.append(Surface.ROTATION_0,90);ORIENTATIONS.append(Surface.ROTATION_90,0);ORIENTATIONS.append(Surface.ROTATION_180,270);ORIENTATIONS.append(Surface.ROTATION_270,180);}/** * Get the angle by which an image must be rotated given the device's current * orientation. */@RequiresApi(api=Build.VERSION_CODES.LOLLIPOP)privateintgetRotationCompensation(StringcameraId,Activityactivity,Contextcontext)throwsCameraAccessException{// Get the device's current rotation relative to its "native" orientation.// Then, from the ORIENTATIONS table, look up the angle the image must be// rotated to compensate for the device's rotation.intdeviceRotation=activity.getWindowManager().getDefaultDisplay().getRotation();introtationCompensation=ORIENTATIONS.get(deviceRotation);// On most devices, the sensor orientation is 90 degrees, but for some// devices it is 270 degrees. For devices with a sensor orientation of// 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.CameraManagercameraManager=(CameraManager)context.getSystemService(CAMERA_SERVICE);intsensorOrientation=cameraManager.getCameraCharacteristics(cameraId).get(CameraCharacteristics.SENSOR_ORIENTATION);rotationCompensation=(rotationCompensation+sensorOrientation+270)%360;// Return the corresponding FirebaseVisionImageMetadata rotation value.intresult;switch(rotationCompensation){case0:result=FirebaseVisionImageMetadata.ROTATION_0;break;case90:result=FirebaseVisionImageMetadata.ROTATION_90;break;case180:result=FirebaseVisionImageMetadata.ROTATION_180;break;case270:result=FirebaseVisionImageMetadata.ROTATION_270;break;default:result=FirebaseVisionImageMetadata.ROTATION_0;Log.e(TAG,"Bad rotation value: "+rotationCompensation);}returnresult;}
Then, pass the
media.Imageobject and the rotation value toFirebaseVisionImage.fromMediaImage():Kotlin
valimage=FirebaseVisionImage.fromMediaImage(mediaImage,rotation)
Java
FirebaseVisionImageimage=FirebaseVisionImage.fromMediaImage(mediaImage,rotation);
- To create a
FirebaseVisionImageobject from a file URI, pass the app context and file URI toFirebaseVisionImage.fromFilePath(). This is useful when you use anACTION_GET_CONTENTintent to prompt the user to select an image from their gallery app.Kotlin
valimage:FirebaseVisionImagetry{image=FirebaseVisionImage.fromFilePath(context,uri)}catch(e:IOException){e.printStackTrace()}
Java
FirebaseVisionImageimage;try{image=FirebaseVisionImage.fromFilePath(context,uri);}catch(IOExceptione){e.printStackTrace();}
- To create a
FirebaseVisionImageobject from aByteBufferor a byte array, first calculate the image rotation as described above formedia.Imageinput.Then, create a
FirebaseVisionImageMetadataobject that contains the image's height, width, color encoding format, and rotation:Kotlin
valmetadata=FirebaseVisionImageMetadata.Builder().setWidth(480)// 480x360 is typically sufficient for.setHeight(360)// image recognition.setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21).setRotation(rotation).build()
Java
FirebaseVisionImageMetadatametadata=newFirebaseVisionImageMetadata.Builder().setWidth(480)// 480x360 is typically sufficient for.setHeight(360)// image recognition.setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21).setRotation(rotation).build();
Use the buffer or array, and the metadata object, to create a
FirebaseVisionImageobject:Kotlin
valimage=FirebaseVisionImage.fromByteBuffer(buffer,metadata)// Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)
Java
FirebaseVisionImageimage=FirebaseVisionImage.fromByteBuffer(buffer,metadata);// Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
- To create a
FirebaseVisionImageobject from aBitmapobject:The image represented by theKotlin
valimage=FirebaseVisionImage.fromBitmap(bitmap)
Java
FirebaseVisionImageimage=FirebaseVisionImage.fromBitmap(bitmap);
Bitmapobject must be upright, with no additional rotation required.
Get an instance of
FirebaseVisionCloudLandmarkDetector:Kotlin
valdetector=FirebaseVision.getInstance().visionCloudLandmarkDetector// Or, to change the default settings:// val detector = FirebaseVision.getInstance()// .getVisionCloudLandmarkDetector(options)
Java
FirebaseVisionCloudLandmarkDetectordetector=FirebaseVision.getInstance().getVisionCloudLandmarkDetector();// Or, to change the default settings:// FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance()// .getVisionCloudLandmarkDetector(options);
Finally, pass the image to the
detectInImagemethod:Kotlin
valresult=detector.detectInImage(image).addOnSuccessListener{firebaseVisionCloudLandmarks-> // Task completed successfully// ...}.addOnFailureListener{e-> // Task failed with an exception// ...}
Java
Task<List<FirebaseVisionCloudLandmark>>result=detector.detectInImage(image).addOnSuccessListener(newOnSuccessListener<List<FirebaseVisionCloudLandmark>>(){@OverridepublicvoidonSuccess(List<FirebaseVisionCloudLandmark>firebaseVisionCloudLandmarks){// Task completed successfully// ...}}).addOnFailureListener(newOnFailureListener(){@OverridepublicvoidonFailure(@NonNullExceptione){// Task failed with an exception// ...}});
Get information about the recognized landmarks
If the landmark recognition operation succeeds, a list ofFirebaseVisionCloudLandmark objects will be passed to the success listener. Each FirebaseVisionCloudLandmark object represents a landmark that was recognized in the image. For each landmark, you can get its bounding coordinates in the input image, the landmark's name, its latitude and longitude, its Knowledge Graph entity ID (if available), and the confidence score of the match. For example:Kotlin
for(landmarkinfirebaseVisionCloudLandmarks){valbounds=landmark.boundingBoxvallandmarkName=landmark.landmarkvalentityId=landmark.entityIdvalconfidence=landmark.confidence// Multiple locations are possible, e.g., the location of the depicted// landmark and the location the picture was taken.for(locinlandmark.locations){vallatitude=loc.latitudevallongitude=loc.longitude}}
Java
for(FirebaseVisionCloudLandmarklandmark:firebaseVisionCloudLandmarks){Rectbounds=landmark.getBoundingBox();StringlandmarkName=landmark.getLandmark();StringentityId=landmark.getEntityId();floatconfidence=landmark.getConfidence();// Multiple locations are possible, e.g., the location of the depicted// landmark and the location the picture was taken.for(FirebaseVisionLatLngloc:landmark.getLocations()){doublelatitude=loc.getLatitude();doublelongitude=loc.getLongitude();}}
Next steps
- Before you deploy to production an app that uses a Cloud API, you should take some additional steps to prevent and mitigate the effect of unauthorized API access.
