OpenCV: unsure how to proceed

OpenCV https://opencv.org/ is the standard open-source library that everyone uses for image detection. It's written in C++.

I want to use it from my .NET UWP apps, but I'm struggling to figure out how. If anyone has solid guidance on how to proceed, I'd love to know! This blog post is just a documentation of my first tentative investigations. (Also eventually I want to use OpenCV from my cross-platform .NET Core apps as well, so I can write an ASP.NET5 webservice running on Linux that does face detection).

I think there should ideally be three layers of NuGet packages:

1. OpenCV.dlls - there should just be a base NuGet package whose only job is to carry the native C++ dlls. It needs to carry the three flavors, x86 + x64 + ARM. It should use the new features in NuGet3 so it can carry all three flavors of dll in a single NuGet package, and at build-time the project.json should pick out the right binaries.
   • There's a catch. The normal "opencv_world300.dll" is huge at 30mb and includes absolutely all OpenCV functionality. Typically my store apps will only need a small subset of that functionality. I see that the "OpenCV.NET" NuGet package instead delivers it split up into 19 separate DLLs, each of them about 1-2mb. It would be best if you could get the DLLs also through a load of smaller NuGet packages, one by one, to pick out which of those binaries you need.
   • Note that a C++ DLL that you built for an earlier version of Windows Store or Windows Phone will not work in UWP. That's because even though UWP supports all the same APIs that were there in the earlier WinRT platforms, it divides them up into different "api-sets", so the binaries are incompatible.
   • Ideally this NuGet package should also carry the native C++ dlls for Linux and for OSX, so it can be used on other .NET Core platforms. I don't know how native dlls are carried on those platforms
      
2. OpenCV.interop - there should be an intermediate NuGet package whose job is solely to contain p/invoke declarations for those DLLs without any fancy wrapping.
   • Maybe it would also include header files, for consumption by C++
   • I don't know how p/invoke declarations work for cross-platform .NET Core.
   • Why decouple this from the raw DLLs? solely to enable the "pick-and-choose" idea above. Maybe it's not worth it. Maybe the "OpenCV.dlls" and "OpenCV.interop" should be combined.
      
3. OpenCV.Net - there should be a higher-level set of wrapper classes, more .NET friendly.
   • Maybe this package need never be written, if everyone is happy using the unwrapped versions of the APIs.
   • Maybe there will be multiple competing versions of this package, with different API designs, some more friendly than others.

 

OpenCV itself 

https://opencv.org/ - the main website has a "download binaries for Windows" link. This includes x64+x86 builds of "opencv_world300.dll" (32mb). I don't know if they're built for UWP but I suspect not. It has two 8.1-era Phone+Store sample apps but they didn't build out of the box. It has a directory called "sources\platforms\winrt" which has instructions for building on WinRT, but these instructions don't include x64, and I suspect they date from before UWP.

https://github.com/Itseez/opencv - the main GitHub repository for OpenCV

https://msopentech.com/blog/2015/03/06/opencv-enabled-for-windows-runtime-family/ - has pre-UWP instructions on how to build binaries from the main GitHub repository using CMake

https://www.nuget.org/packages/OpenCV/ - this NuGet package with 18k downloads claims to contain "binaries and build targets". This isn't true. It contains C++ header files, and it contains a .targets file which (based on build parameters) pick up the OpenCV native DLLs that it assumes you got from elsewhere and copied into a particular location. It only does this for x86 and x64; not ARM.

There are two old sites - https://github.com/Microsoft/opencv was the old staging repo where WinRT support was first added to the main OpenCV GitHub, and https://code.opencv.org/projects/opencv/wiki/WindowsRT had old instructions on how to build for WinRT. I think both links are out of date.

Analysis: I suspect there are no "ready-to-go" binaries here for UWP, nor any useful NuGet packages. If I want to use OpenCV from source, I'll have to learn the thing's build system, configure it to build UWP, and built it myself. I'll have to write a OpenCV.dlls NuGet package from scratch myself. I don't know whether the work I do here can be submitted back into OpenCV's daily builds, or if I'll have to keep it separate.

 

OpenCV Extension SDK 

https://channel9.msdn.com/Events/Build/2015/3-82 - this video discusses an "OpenCV for Windows Store" Extension SDK which contains native binaries for UWP apps. However, the Extension SDK has never been publically released. (Also, ExtensionSDKs were used in the past because prior versions of NuGet weren't good at selecting x64/x86/ARM binaries based on build-time configuration. That limitation has gone away, and now NuGet is a much better option than ExtensionSDK).

Analysis: The author of the video has moved on to a different team. I've contacted his former colleagues to see if I can get anywhere with this. I don't know if it will be a single huge 30mb "world" dll, or a series of smaller ones.

 

OpenCVSharp

This NuGet package contains a 30mb "world" binary for x86+x64 (not ARM) that come from here. It also contains C# wrappers that are as close as possible to the raw C API. This is the most widely used package, with 32k downloads total.

I'm confused, because the NuGet package has that huge DLL, but then the project's install.ps1 seems to have them split up into separate pieces.

The NuGet package only installs onto desktop .NET Framework 4.0 and higher -- not .NETCore, and not UWP.

As for the raw native binaries themselves, I don't know. But given the desktop focus, and the lack of ARM, I strongly suspect they won't work on UWP.

https://github.com/shimat/opencvsharp - this is the main repository.

The OpenCVSharp NuGet is actually offered in four flavors:

1. https://www.nuget.org/packages/OpenCvSharp3-AnyCPU/ - the main one for OpenCV3 which was released in 2015; most recent
2. https://www.nuget.org/packages/OpenCvSharp3-WithoutDll/ - as above, but missing the "world.dll" (so you'll pick it up yourself elsewhere)
3. https://www.nuget.org/packages/OpenCvSharp-AnyCPU/ - like [1] but for OpenCV 2.4
4. https://www.nuget.org/packages/OpenCvSharp-WithoutDll/ - like [2] but for OpenCV 2.4

There's also this NuGet package https://www.nuget.org/packages/OpenCVSharp/ but it doesn't seem to be related.

Analysis: this project seems to provide the correct "interop p/invoke" layer. But I think it won't be a good source of binaries for UWP.

 

OpenCV.NET

This NuGet package has a nicer .NET-style set of wrapper classes. I don't understand how/where the NuGet package picks up its raw binaries from, but it appears to get them somehow! The binaries seem to be x86 only (not x64, not ARM) and are split into ~15 smaller DLLs. The package has been downloaded 8k times. The most recent update was 9 months ago.

The NuGet package only works on desktop .NET Framework 4.0 and higher (not UWP).

https://www.nuget.org/packages/OpenCV.Net/

Analysis: I think it'd be ideal to split out the "interop p/inoke" layer from the wrapper classes. I'm interested to know whether the interop declarations for "split-into-separate-dlls" that this project uses is different from the interop declarations for "world.dll". In any case, this project isn't a suitable source for raw binaries.

 

Emgu

This project does provide ARM and WinRT for 8.1-era. However I haven't read anything to indicate support for UWP. Its NuGet package was last updated 20 months ago and has 6k downloads.

This project requires a commercial license for ARM and WinRT.

https://www.emgu.com/wiki/index.php/Main_Page

https://www.nuget.org/packages/VVVV.EmguCV/

Analysis: this isn't in the OSS spirit that I like about .NET Core, so I haven't pursued it.

 
 

Stack Overflow 

https://stackoverflow.com/questions/14065370/using-opencv-in-a-windows-store-app-with-visual-c - this seems to be the most valuable StackOverflow resource. It has Raman Sharma (author of the "OpenCV for Windows Store Apps" ExtensionSDK) give some answers on how he built OpenCV for WinRT.

Comments

• Anonymous
  November 26, 2015
  Nice article on Open CV

• Anonymous
  December 06, 2015
  Thanks for all your effort. I also want to use OpenCV in my C# UWP app for Sudoku puzzles scanning/detection. I will keep following your progress.

• Anonymous
  December 23, 2015
  This is great.....I thought I was confused but your analysis just confirms there is still a gap to fill especially regarding building for ARM processors..