Creating something from nothing, asynchronously [Developer-friendly virtual file implementation for .NET improved!]
**
This blog has moved to a new location and comments have been disabled.
All old posts, new posts, and comments can be found on The blog of dlaa.me.
See you there!
Comments
Anonymous
November 04, 2009
Would it be possible to ensure that the StreamContents callback was called on the UI thread (or at least when the UI thread wasn't blocked). I'd like to put a screenshot of some of the UI into the virtual file, but of course I can't access any WPF elements from anything but the UI thread. Grabbing the bitmap before calling DoDragDrop works, but of course that's means we always take the perf hit. Maybe I should stop worrying. I'd probably want to (eventually) use the bitmap as visual feedback for the DnD operation anyway.Anonymous
November 05, 2009
swythan, I considered trying to do that. But then I decided that was probably out of scope for VirtualFileDataObject because there's no way of knowing what the developer intends to do in their begin/end actions. Like I mentioned in the article, an MVVM approach is probably best here - and for that there's usually no need to be on the UI thread. Therefore, it seemed like forcing the issue in VirtualFileDataObject would not really be addressing the 80% case very well. And because it's so easy for the developer to get to the UI thread, I chose the current approach for its simplicity and generality. I'd expect that you can do what you want within in the current framework (perhaps via Dispatcher.BeginInvoke), but if you find that you can't, that would be great feedback to have. Thanks for taking the time to write - I hope this is helpful!Anonymous
November 11, 2009
How about stroring the SynchronizationContext.Current in the ctor instead of a dispatcher? There could be a ctor parameter or property to control if it should be used. That way the the user can tell the object to either marshall callbacks or not and the object doesn't need a reference to any dispatcher.Anonymous
November 12, 2009
onovotny, The use of Dispatcher in the example above is part of the sample application, not part of the VirtualFileDataObject control; I have specifically tried to keep threading issues out of VirtualFileDataObject. I'm not sure from reading your suggestion whether you're proposing a change to the sample or to VirtualFileDataObject itself. If you're just talking about the sample, your suggestion sounds like it would work as well. :) I think the current way is a bit clearer for a sample, but your way is probably a more practical solution.Anonymous
November 13, 2009
...having to create your own IDragSource. this also provides access to the drag drop visuals feature demonstrated in the Win7 SDK DragDropVisuals sample http://code.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=shellapplication&DownloadId=6769 there is a managed version of this as well found here http://blogs.msdn.com/adamroot/pages/shell-style-drag-and-drop-in-net-wpf-and-winforms.aspxAnonymous
November 13, 2009
messed up my post above... the title meant to be "use SHDoDragDrop to avoid..."Anonymous
November 13, 2009
Chris, That's a great suggestion! In my case, IDragSource was pretty easy to write, so I don't feel too, too silly. :) But you're right that folks should be aware of the SHDoDragDrop API because it seems like another good option: http://msdn.microsoft.com/en-us/library/bb762151(VS.85).aspx Thanks for writing!Anonymous
July 20, 2010
The comment has been removedAnonymous
July 20, 2010
Tyler, Folks report that problem to me every now and then and so far it has always been because they're running under the debugger - if they run without debugging (Ctrl-F5) instead, it works just fine. Also, I'm pretty sure hitting F5 to continue after the exception (which is handled) also works fine. Maybe VS 2010 has a different setting for catching first chance exceptions, or maybe everyone's just forgotten that they disabled them in VS 2008 a long time ago. :) Please give that a try and let me know if you're still having trouble. Thanks!Anonymous
August 05, 2010
Hey Delay! Great blog! This is exactly what I was looking for; I need to support dragging and dropping of virtual files which need to first be transferred from a hardware device via a slow serial connection. I also get the exception noted above in VS2010 it's caused by the call to Marshal.ThrowExceptionForHR( ) method in theVirtualFileDataObject::GetData( ) method. It seems to result in an InteropServices.COMException exception being thrown which for whatever reason doesn't get caught by the exception handler. You can disable break on user unhandled System.Runtime.InteropServices.COMException in the Debug | Exceptions dialog to stop this.Anonymous
August 05, 2010
Delay, Just out of interest would it be possible to implement a graceful way of cancelling an asynchronous virtual file transfer?Anonymous
August 06, 2010
The comment has been removedAnonymous
March 06, 2011
Hi, great piece of code! Now I have a problem: can current code works for multiple files? I saw your previous article has 1 "You can drag multiple virtual objects, you know" article, but the native language is scary for me. Any shortcuts? Much appreciate.Anonymous
March 07, 2011
ok, so I found out that SetData can accept an array of DataObject. problem solved. Thanks for the code anyway.Anonymous
March 08, 2011
vhanded, Glad you got that worked out!Anonymous
April 19, 2011
Thank you very much for this article. It is very well written and thought out. I am working on code that accepts a Drop from the VirtualFileDataObject class. I have figured out how to make a GetData call with FileGroupDescriptorW and parse out the array of FileDescriptor structures that is returned. I am having trouble figuring out how to get the corresponding file data. I've tried making a GetData call with "FileContents" but all I get back is null. Can you give me any ideas how to do this?Anonymous
April 19, 2011
Scott Myers, Wow - it's been a while since I wrote this code - and I didn't need to handle the receiving end at the time... :) That said, as I look at the code now, it appears that the "FileContents" data may be special because it's indexed (FORMATETC.lindex != -1) because there may be multiple representations of the same data. So my suggestion would be to check the call you're making to GetData and be sure it's configured correctly for this. Alternatively, maybe try handling a file drop from something "real" like Windows Explorer just in case I've overlooked something in my code. Thanks for the kind words - I hope this helps! :)Anonymous
October 14, 2011
This is a fantastic piece of work and exactly what I have been trying to assemble. I cannot seem to get this ported over to VB.NET - what are the chances of seeing a version of this code in VB? (saying that is like a Hail Mary pass)Anonymous
October 15, 2011
John Lamberis, Thanks for the kind words! :) Regarding a full VB port, I'm afraid that's not something I feel likely to get to given how many other things are on my list. However, if you've got it mostly done and have one or two specific questions, I may be able to help. And if you get the whole thing ported, I'd be happy to link to your work so others could use it! But you might also consider simply compiling my C# implementation into an assembly and then referencing that assembly from your VB.NET project and using it just like any other .NET assembly. Aside from sticking this code in its own assembly, you shouldn't need to care that it's C# as that will all be hidden away in its own assembly. Just a thought... :)Anonymous
January 27, 2012
Thanks for this code; it's extremely helpful. One improvement I have to suggest: implement IStreamWrapper.SetLength to set the size of the IStream, like so: /// <summary> /// Sets the length of the current stream. /// </summary> /// <param name="value">The desired length of the current stream in bytes.</param> public override void SetLength(long value) { _iStream.SetSize(value); } When using the VirtualFileDataObject to transfer very large files, setting the length beforehand to a reasonable size makes an enormous difference in speed--else IStream.Write can end up being unacceptably slow. (Of course, it's a tradeoff with memory usage...)Anonymous
January 27, 2012
Matt Winckler, Thanks, Matt, this sounds like a great suggestion! I've added it to my TODO list to investigate if/when I revisit this topic.Anonymous
April 25, 2012
David, great work. Unfortunately though I have noticed that the operation occurs synchronously when dragging a virtual file into an Outlook message (drag and drop to windows explorer occurs asynchronously as expected). As this is the case in your sample I don't believe this to be due to my implementation, do you have any ideas as to why this is or how it might be avoided? ThanksAnonymous
April 26, 2012
Nick, Thanks! Regarding the behavior you're seeing, what I would do first is check the value of the IsAsynchronous variable when the synchronous operation is started. IsAsynchronous defaults to true, but Outlook may be calling IAsyncOperation.SetAsyncMode to set it to false and that would lead to the behavior you're seeing, I think.Anonymous
June 28, 2012
How do I implement this with drag and drop within a WPF application? My app need to drag to the File Explorer and drag items with in itself. I can't figure out how to make VirtualFileDataObject.DoDragDrop work with my existing drag drop code.Anonymous
June 28, 2012
Steve, If the drag source and destination are both within the same application, can you detect that and bypass the logic here to manage the entire operation internally? That's probably what I'd try (perhaps via a custom drag object or clipboard format?), though I haven't actually done so myself and may be oversimplifying. :)Anonymous
August 28, 2012
I'm trying to do the same as Steve: Drag an object either to the desktop or within the application. The problem is that i can do nothing with the virtualfiledataobject inside the drop handler of the application because i cannot convert the drop args there into the neccessary object. Also i cannot detect when i drag the object, where it would be dropped on: onto the explorer or onto the application. I have tried to call windows dragdrop and virtualfiledataobject drag drop at the same time but that does not work also.Anonymous
August 29, 2012
Joe, Let's deal with one thing at a time. You say you are having trouble dragging things to the desktop, but my sample application seems to work fine for that. Try dragging either of the bottom two items onto the desktop - a file will be created with the appropriate contents. Is this not working for you, or is there more to it?Anonymous
August 29, 2012
Question, Is there a way to support dragging and dropping from the WPF application to another app (not windows explorer) such as Chrome, Evernote, etc.? I have been half successful with my question since I am able to drag to the Windows Live Mail application and Outlook for attachment, but those are the only two that have worked for me.Anonymous
August 29, 2012
Chevon, The sample application already demonstrates dropping into another application. :) Dragging the "Text and URL" item onto Chrome or Internet Explorer will open that URL in the browser. (Dragging the "Text only" item onto Chrome's address bar will initiate a search for that text.)Anonymous
August 30, 2012
David, Thanks for the tips. Worked great! However, I would like to go a little bit further and allow files (virtual or not) to be dragged to Chrome/Gmail, as is done with Windows Explorer. Do you know the steps I can take to achieve this? I believe I need to specify the FileDrop DataFormat, along with an array of strings/filenames. However, your code does not current accept this format.Anonymous
August 30, 2012
Chevon, What you suggest around using DataFormats.FileDrop sounds about right to me. I'd expect that could be added to the sample implementation to get the effect you want.Anonymous
September 05, 2012
David, Loving this work. I'm using it to rename existing files as they are "dropped" into a new location. It's sluggish for large files using the following code: new FileDescriptor { Name ="SOME NEW NAME" StreamContents =stream=>{ FileStream fs = File.OpenRead(actualFilePath); fs.CopyTo(stream); fs.Close(); } } Can you think of a faster alternative? SteveAnonymous
September 06, 2012
woodced1979, What you're doing looks pretty efficient to me. :) One thing you might do is wrap the FileStream in a "using" which would ensure it's Dispose-ed and also avoid the need for you to call Close. Something else you might consider is doing a P/Invoke out to the CopyFile Windows API because that may be able to perform the copy a bit more efficiently.Anonymous
September 06, 2012
David, Thanks for the response. I'm finding that a 46Mb file takes a minute or so versus a second or so using the built in System.Windows.DataObject.. I'm not sure I quite understand you with regard to the CopyFile Windows API method. Where abouts do you think I would implement this?Anonymous
September 07, 2012
woodced1979, I'm suggesting that calling msdn.microsoft.com/.../aa363851(v=vs.85).aspx instead of OpenRead/CopyTo may be more efficient because it allows the OS to do the work for you and it has all kinds of resources that stream read/write can't take advantage of. That said, it's going to take time to write 46Mb to disk, so this is unlikely to be instantaneous. The speed you're seeing from DataObject may be a result of it taking some shortcuts of its own (ex: if the data is in memory, it doesn't need to be read from disk).Anonymous
September 24, 2012
I have a list of objects. I use .net's DragDrop.DoDragDrop to drop an item of that list to another list inside my application. In the drop handler of the list I drop the item on, I get an object of the list item's type and everything is ok. Now I want to drop that item also on the desktop. So I use the VirtualFileDataObject's dragdrop operation to do so. This works also without problems. The problem is that I want to be able to drop the item on the other list AND to the desktop. I do not know how to manage this. When I use the dragdrop operation of the Virtualfiledataobject, I get an object of type FileGroupDescriptorW and FileContents in the list's drop handler. How can I get the item out of that?Anonymous
September 25, 2012
Joe, You can register multiple types with IDataObject This is how the sample app supports text+URL+file. My thinking is that you can try to include a custom type which contains the info to get back to your list item. (It probably can't be the item itself because of the native/managed transition, but an index should be just fine.) Hope this helps!Anonymous
September 25, 2012
Thank you David, it works! I just serialized my object as byte array, just as you did in the example and defined an own DropType so I can identify my object. With this I can deserialize the byte array in the OnDrop Handler of my list so I can get my object. Great Work!Anonymous
November 22, 2012
By me it crashes with exception COMException was unhandled by user code, Error in the structure FORMATETC (Exception HRESULT: 0x80040064 (DV_E_FORMATETC)). FORMATETC is taken from namespace System.Runtime.InteropServices.ComTypes. Is there easy way to fix it?Anonymous
November 22, 2012
Alexandre N, This is a debugger configuration issue. Please see Tyler's similar comment above: blogs.msdn.com/.../creating-something-from-nothing-asynchronously-developer-friendly-virtual-file-implementation-for-net-improved.aspx And my response: blogs.msdn.com/.../creating-something-from-nothing-asynchronously-developer-friendly-virtual-file-implementation-for-net-improved.aspxAnonymous
February 05, 2013
David, Great article that has really helped me - thanks a lot :) I'm finding a potential scale issue. The stream writes for the VirtualFileDataObject take progressively longer as the file is written. I am writing up to 128kb to the stream with each write. The first few writes take about 8ms, but after 200MB or so they are taking more like 90ms. This means a 20MB file is written almost instantly whereas a 200MB file takes several minutes - much more than the expected 10x longer. Do you know why this may be? ThanksAnonymous
February 06, 2013
Tom Colvin, Glad you liked it! Regarding the behavior you're seeing, I'm not sure what's going on offhand. My guess based on how you describe it is that something is trying to buffer the entire stream as it's being written. (I looked again at my code briefly and it doesn't seem like it would be guilty of doing that.) It might be interesting to step through the code or profile the memory use of your app in this scenario or even try a simple (yeah, I know this code isn't simple...) native code test app to check if it has the same behavior. Hope this helps!Anonymous
April 25, 2013
How can I get Dropping location?Anonymous
April 26, 2013
I don't see that you can - it's likely to be in a different process about which the drop source knows nothing, so it's not clear what kind of information could be provided to the source that would be generally useful.Anonymous
May 21, 2013
Hi David, Really a great article, thanks a lot!!! I am facing one issue, how to force synchronous process to asynchronous? My use case is - I am dragging file to Outlook in that case VirtualFileDataObject doesn't call the GetAsyncMode() function, like it get called when dragging to explorer. I've tried all option to set IsAsynchronous to true, but not succeed. Waiting for your valuable response. Thank you.Anonymous
May 22, 2013
Vijay G, I'm not sure that's possible. If the target application doesn't want to opt-into asynchronous mode, I think that's its choice and that Windows doesn't try to shim the operation. As a drag-drop source, you should try to support both modes, but be happy to provide data however the target asks for it. :)Anonymous
May 22, 2013
Thank you David for your valuable comment.Anonymous
May 22, 2013
Hi, I have a problem when trying to drag/drop large files using VirtualFileDataObject. What I do is to open a filestream to the source file, reading the bytes from the source filestream and writing those bytes to the Stream, which was delivered by the callback function of "fileDescriptor.StreamContents", using the "stream.Write" function. But this seems to write all bytes from the source file to memory first, before dumping it all to the harddisc. This sometimes leads to an OutOfMemoryException if the file is bigger than several hundered Megabyte. Is there any solution to this? I have to admit that I'm not that skilled in COM programming, so I do not know what to change in the VirtualFileDataObject...Anonymous
May 23, 2013
Joe, It sounds like you're providing an implementation of FileDescriptor.StreamContents that reads from a file and writes it to a stream. When doing so, you want to read from the file in a series of small chunk and write each chunk to the stream before moving on to the next chunk. For example, create a 1000 byte buffer, read into it from the file, write that out to the stream, read the next 1000 bytes, etc.. In this manner, you'll only be using 1000 bytes of memory at a time no matter how large the file is. This is such a common pattern that recent versions of .NET offer the Stream.CopyTo method (msdn.microsoft.com/.../system.io.stream.copyto.aspx) which handles this for you! I hope this helps!Anonymous
May 23, 2013
The comment has been removedAnonymous
May 23, 2013
Sorry, chunks of 4096 bytes, not kb :)Anonymous
May 24, 2013
Joe, Got it, thanks for clarifying! Unfortunately, you're working right at the boundary of my memory/experience, so I don't know the right answer. :( That said, I think you're on the right track - what I'd suggest is to look at the SetData method that calls CreateStreamOnHGlobal and see if there are other options (ex: dwAspect or tymed) that would allow you to do what you want (i.e., provide something other than an in-memory stream). Also, I think there's a deferred-render mode for clipboard data that might be relevant/useful here because it waits to provide the data until it's needed and the file name should be available at that time. If you figure this out, please let me know what you ended up doing - I'd love to learn! :)Anonymous
January 28, 2014
Hi David, Thank you for this article, I've finally found a good way to drag and drop stuff from my .net app. I have a question though. I'm using the .net 35 framework and in my c# app (complied with the "any cpu" option) I can't get the VirtualFileDataObject class to be initialized. I saw that errors occure when initializing the static properties like below: private static short FILECONTENTS = (short)(DataFormats.GetDataFormat(NativeMethods.CFSTR_FILECONTENTS).Id); With the debugger I saw that the value of Id returned is over the maximum value for a short. For instance, if I do: var d = System.Windows.DataFormats.GetDataFormat("FileGroupDescriptorW"); d then equals 0xc11e thus casting it to a short raises an OverflowException. I'm using windows 7 x64. Is it a know issue? What can I do to fix that? Thank you in advance,Anonymous
January 28, 2014
Matthieu, This is not a known issue and I'm not seeing the same behavior on my machine with the sample application above. I do see Id values greater than 0x8000 and even with that the cast shown above succeeds. My guess is that your project has the /checked compiler option enabled: msdn.microsoft.com/.../h25wtyxf.aspx If so, you should be able to run the relevant code in an unchecked { ... } block or change the types from short to int to avoid the OverflowException.Anonymous
February 03, 2014
Hi David, Thank you! I removed the "checked" flag and it works now like a charm :) However I was wondering whether we could show the windows copy file form while dropping a large file. I'm trying to extract a 650mb file and nothing appears during the transfert. As you did in your improved version I can handle this in my application but the user won't be able to see the remaining time, right?Anonymous
February 04, 2014
For those that do not want the entire file to be cached in memory while being dropped, you have to wrap your stream into an IStream: /// <summary> /// Simple class that exposes a read-only Stream as a IStream. /// </summary> private class StreamWrapper : IStream { private Stream _stream; public StreamWrapper(Stream stream) { _stream = stream; } public void Read(byte[] pv, int cb, System.IntPtr pcbRead) { Marshal.WriteInt32(pcbRead, _stream.Read(pv, 0, cb)); } public void Seek(long dlibMove, int dwOrigin, System.IntPtr plibNewPosition) { Marshal.WriteInt32(plibNewPosition, (int)_stream.Seek(dlibMove, (SeekOrigin)dwOrigin)); } public void Clone(out IStream ppstm) { throw new NotImplementedException(); } public void Commit(int grfCommitFlags) { throw new NotImplementedException(); } public void CopyTo(IStream pstm, long cb, IntPtr pcbRead, IntPtr pcbWritten) { throw new NotImplementedException(); } public void LockRegion(long libOffset, long cb, int dwLockType) { throw new NotImplementedException(); } public void Revert() { throw new NotImplementedException(); } public void SetSize(long libNewSize) { throw new NotImplementedException(); } public void Stat(out System.Runtime.InteropServices.ComTypes.STATSTG pstatstg, int grfStatFlag) { throw new NotImplementedException(); } public void UnlockRegion(long libOffset, long cb, int dwLockType) { throw new NotImplementedException(); } public void Write(byte[] pv, int cb, IntPtr pcbWritten) { throw new NotImplementedException(); } } Then, in the SetData method, change its signature: public void SetData(short dataFormat, int index, Stream stream) { ... var iStream = new StreamWrapper(stream); ... // Ensure the following line is commented out: //Marshal.ReleaseComObject(iStream); return new Tuple<IntPtr, int>(ptr, NativeMethods.S_OK); ... } By doing so, while extracting/dropping large file, you don't have a new memory stream which contains the data! Hope it helpsAnonymous
February 04, 2014
Matthieu, Thanks a lot for the example above for in-memory caching! Unfortunately, I don't know of a way to reuse the Windows copy dialog in this scenario.Anonymous
February 05, 2014
I've found it -> msdn.microsoft.com/.../ff362447.aspx var FILEDESCRIPTOR = new NativeMethods.FILEDESCRIPTOR { cFileName = fileDescriptor.Name, dwFlags = NativeMethods.FD_ATTRIBUTES | NativeMethods.FD_SHOWPROGRESSUI, }; And now we have the windows copy dialog ;)Anonymous
February 05, 2014
Matthieu, Awesome, thanks for sharing! :)Anonymous
March 18, 2014
David, great article! One question, has anyone tried Matthieu's post on using a IStream? I ran into some problems after changing the signature of the SetData Method. Other methods that call it have issues then. Matthieu's post on the windows copy was cool!