Creating intelligent apps with Computer Vision

After some experimenting with SkiaSharp, I came up with the idea to build a game like Draw It. Normally you draw something and someone needs to guess what you’ve drawn. This time I wanted to take a bit of a different approach. I wanted to draw something with SkiaSharp and put Computer Vision to the test!

Computer Vision is (a small) part of the Cognitive Services that Microsoft offers. The Cognitive Services allow developers without knowledge of machine learning to create intelligent applications. There are 5 different categories of cognitive services (at the moment of writing):

• Vision: Image and video processing.
• Knowledge: Mapping complex data to use for recommendations or semantic search.
• Language: Process natural language to recognise sentiment and determine what users want.
• Speech: Speaker recognition and real-time translation.
• Search: More intelligent search engine.

The categories have a lot more to offer than described above. If you want to know more, please visit the Cognitive Services website. There is a 30-day trial available if you want to try it!

Computer Vision 

Computer Vision API is all about analysing images and videos. After sending an image or video, the API returns a broad set of metadata on the visual aspects of the file.  Besides telling you what’s in the picture, it can also tell who’s in the picture. Because it’s not trained to recognize everybody on this planet, this will only work with celebrities. If you’re not recognized by Computer Vision, it can still tell a lot about your picture. It will add tags, captions and even guess your age! It’s really fun to experiment with!

Computer Vision also supports Optical Character Recognition (OCR) what can be used to read text from images. OCR is very useful when you’re building apps that need to interpret paper documents. Handwritten text are still in preview.

If you want to test if Computer Vision suits your needs, you can give it a try on the website without having to code a single line. This demo will only work for basic scenarios (upload an image), if you want to use the more advanced features (like real-time video), I’d suggest you try the API directly.

Getting started

Using Cognitive Services is pretty straightforward because all of the magic is available through REST API’s. For this post I will focus on the Computer Vision API, but the other APIs work in a similar way. You can use the API’s with the following steps:

1. To start using Cognitive Services you can create a trial account on the website. Click the “Try Cognitive Services for free” button and then click “Get API Key” for the Computer Vision API. After signing in with your account you will get an API key that allows you to create 5000 transactions, limited to 20 per minute. Note: the trial is limited to the westcentralus API (https://westcentralus.api.cognitive.microsoft.com/vision/v1.0).
   If you want to implement Cognitive Service in your production environment, you can obtain an API key from the Azure Portal. Therefor you need to create an Cognitive Services resource of the type you want to use (for example: Computer Vision API). When the resource is created, you can get your keys from the Quick Start -> Keys.
2. Because Computer Vision is exposed as an REST API, you can call it from your app using HttpClient. Instead of creating your HttpClient, you can also use the NuGet package to make your calls to the Computer Vision API.
   Note: In most cases you don’t want to make the calls directly from your app, but call the Cognitive Services from your server. This will prevent users from stealing your API key (and burning your Azure credits) when they decompile your app or intercept your calls.
3. After adding the Microsoft.ProjectOxford.Vision to your backend or frontend, you will be able to use Computer Vision with a few lines of code:

   // If you're using a trial, make sure to pass the westcentralus api base URL
   var visionClient = new VisionServiceClient(API_Key, "https://westcentralus.api.cognitive.microsoft.com/vision/v1.0");
   VisualFeature[] features = { VisualFeature.Tags, VisualFeature.Categories, VisualFeature.Description, VisualFeature.ImageType, VisualFeature. };
   var result = await visionClient.AnalyzeImageAsync(stream, features.ToList(), null); 

   First we create an VisionServiceClient object with an API key and a base URL. Trial keys only work with the WestCentralUS instance and therefor I’m also passing the BaseUrl. When you’ve created a client, you can call AnalyzeImageAsync. This function takes in a Stream (which contains the image) and a list of VisualFeatures. The VisualFeatures specify what you want Computer Vision to analyse. If you prefer to use the HttpClient instead of the NuGet package, I’d recommend the video where René Ruppert explains how to use HttpClient to consume Cognitive Services.

Results

Unfortunately my drawing skills are not good enough for Computer Vision to recognise (with great certainty) what I was drawing. Although it was not sure what I was drawing, it suggested “Outdoor house” with a accuracy of 0.1875. An other tag Computer Vision suggested was “Clock” and I think that’s because the sun looks like a clock.

If you want to give your drawing skills a shot, the code of my experiment is on GitHub. For the drawing part of the application I’ve used some code snippets from the Xamarin.Forms samples. Besides this sample app, Xamarin has a lot of great sample apps on their GitHub.

If you want to learn more about the Computer Vision API or Cognitive Service, make sure to check out the related links below. If you have any questions or suggestions, please don’t hesitate to contact me on Twitter.

Related links:

• Microsoft Azure: Computer Vision
• Xamarin Developer: Cognitive Services
• Blog Xamarin: Building Smart Apps with Microsoft Cognitive Services
• YouTube: Building Your First Intelligent App with Xamarin and Microsoft Cognitive Services
• NuGet: Microsoft.ProjectOxford.Vision
• YouTube: Getting started with Cognitive Services

 

 

 

Drawing with SkiaSharp in Xamarin.Forms

Where most people try to find suitable tools, services and frameworks for the app they are going to build, for me it’s often the other way around. I’m often looking for app ideas to experiment with specific tools, services or frameworks. This time I wanted to give SkiaSharp a try!

In 2005 Google acquired the 2D graphics engine called Skia. The engine is open-source and nowadays used in Google Chrome, Chrome OS, Android, Mozilla Firefox, Firefox OS, and many other products. To extend the use of this engine to the .NET world, Xamarin created a binding called SkiaSharp. This binding can be used in Windows Forms, WPF, UWP, iOS, Android, tvOS, Mac, Xamarin.Forms. SkiaSharp works similar across platforms, but each platform has a few specific classes. This blogpost will focus on Xamarin.Forms since this covers a wide range of platforms and it doesn’t contain a 2D graphics engine itself.

Getting started

1. First, you need to add the SkiaSharp.Views.Forms NuGet package to your Xamarin.Forms PCL (or .netstandard) project and platform specific projects. This package depends on the SkiaSharp package, and therefor this NuGet package will be added as well.
2. After adding the package, you are able to use the SkiaSharp Views in your Xaml or in your code. If you want to use SkiaSharp in your Xaml, you first need to define the namespace:

   xmlns:skia="clr-namespace:SkiaSharp.Views.Forms;assembly:SkiaSharp.Views.Forms"
   

   To use the Views in your code you only need to add the same namespace as a using.

3. The Views.Forms namespace contains a View called SKCanvasView. As the name suggests, this View will create a canvas to draw on. It’s not possible to draw directly onto the SKCanvasView, but the view exposes an EventHandler called PaintSurface. This handler is called when the canvas needs to be (completely) re-drawn and contains an (very important!) argument of type SKPaintSurfaceEventArgs. Typically the event will be raised during initialisation or on rotation. It’s also possible to trigger the event manually by calling the InvalidateSurface() on the SKCanvasView. Registering your event handler can be done in code or in Xaml. In Xaml this can be achieved with the following code:

   <skia:SKCanvasView PaintSurface="Handle_PaintSurface"/>
   

   With this Xaml you tell the SKCanvasView to call the Handle_PaintSurface in your code-behind when the canvas needs to be (re-)drawn. The argument contains a property Surface which on his turn contains a property called Canvas. On this Canvas object we can start drawing!

4. The basics of drawing are pretty simple. You need to specify the coordinates of the object you want to draw and you need to pass an instance of type SKPaint. The SKPaint object defines how your object will look. There are 3 types of styles:
   • Fill: Specify on how to fill an object like for example a circle.
   • Stroke: Specify on how thick and what color the drawing of lines will be.
   • StrokeAndFill: Specifies the styling of the lines drawn and how to fill that object.
5. A few examples of drawing a simple shape:

   SKPaint blueFill = new SKPaint { Style = SKPaintStyle.Fill, Color = SKColors.Blue };
   SKPaint blackLine = new SKPaint { Style = SkiaSharp.SKPaintStyle.Stroke, StrokeWidth = 10, Color = SKColors.Black };
   
   // Draws a line from 0,0 to 100,100
   canvas.DrawLine(0, 0, 100, 100, blackLine);
   
   // Draws a rectangle left=0, top=0, right=300, bottom=150
   canvas.DrawRect(new SKRect(0, 0, 300, 150), blueFill);
   
   // Draws a circle at 100,100 with a radius of 30
   canvas.DrawCircle(100, 100, 30, blackLine);
   
   

Beyond the basics

SkiaSharp has a lot more to offer, but that’s way to much to cover in one blogpost. If you want to take it a step further, the following topics might be interesting:

• Bitmaps allow you to use images in your Canvas.
• With Transformations you can modify the coordinates or size of your objects. This is often used in combination with animations. At this moment you can use the following transformations: Translate, Scale, Rotate and Skew.
• Save() allows you to save the current state of the canvas. With Restore() you can return to the last saved state. This is very useful when you’ve applied Transforms on your Canvas and you want to go back to the previous state.
• Animations can be created by calling InvalidateSurface(). This will trigger the PaintSurface event that you can use to apply your changes to the canvas.
• In a lot of cases you might want to write something on your Canvas and therefor DrawText() was created. Skia allows you to draw text with a lot of styling options.
• If you want to draw a line from point to point (and so on), SKPath might fulfill your needs. Paths can be customised (visually) pretty easy. You can also create curves to smoothen your path.
• Creating a gradient can be easily achieved with the SkShader class. You can combine this with some Noise to create a nice texture.

Conclusion

Drawing with SkiaSharp is pretty easy and fun, especially when using the Xamarin Live Player. The Live Player allows you to edit your code and instantaneously see your changes. This improves the speed of coding a lot!

SkiaSharp is available for a wide range of platforms, which makes it very powerful. Although you can share a lot of code, you always have to keep in mind what devices sizes and device forms you are developing for.

The documentation on the Xamarin website is great and they also have a lot of samples. I’ve created a sample project myself which contains the basics of drawing with SkiaSharp. The result is shown on this image (only tested on iPhone 8 and Nexus 5). You can find the code on GitHub. Feel free to improve the drawing by creating a PR!

If you have any questions, don’t hesitate to contact me through the contact form or on Twitter.

Related links:

• Channel 9 – SkiaSharp Graphics for Xamarin.Forms
• Xamarin –  An Introduction to SkiaSharp
• Xamarin – 2D drawing