Opening files that don't have associated programs

Recently, I was adding the ability to open files from an application at work. Luckily, I had run into this situation before, so I knew that you could run a Process.Start on the file in question and have Explorer open it for you. What I didn’t know was that it would throw a Win32Exception if the file didn’t have an associated program to open it. My next step was what any good developer would do: Google it.

Fairly early on in my searching, I came across a post by Ned Batchelder that described the EXACT same scenario I was working on! Ned wanted to open a file with an unknown extension from a managed application just like me. He details his Google search and explains how he discovered information about the API functions, ShellExecute and ShellExecuteEx. Basically, you can call ShellExecuteEx with an “open” verb and Explorer will attempt to open the file. If it fails with an SE_ERR_NOASSOC error, then call ShellExecuteEx again but using an “openas” verb. Doesn’t sound too bad, though some code snippets still would’ve sped the process up for me :-) I did learn it better this way, though.

Anyway, here’s where I started. First, I pulled in the definition for the SHELLEXECUTEINFO structure.

Friend Structure SHELLEXECUTEINFO
  Public cbSize As Integer
  Public fMask As SEE_MASK
  Public hwnd As IntPtr
  <MarshalAs(UnmanagedType.LPTStr)> _
  Public lpVerb As String
  <MarshalAs(UnmanagedType.LPTStr)> _
  Public lpFile As String
  <MarshalAs(UnmanagedType.LPTStr)> _
  Public lpParameters As String
  <MarshalAs(UnmanagedType.LPTStr)> _
  Public lpDirectory As String
  Dim nShow As Integer
  Dim hInstApp As SE_ERR
  Dim lpIDList As IntPtr
  <MarshalAs(UnmanagedType.LPTStr)> _
  Public lpClass As String
  Public hkeyClass As IntPtr
  Public dwHotKey As Integer
  Public hIcon As IntPtr
  Public hProcess As IntPtr
End Structure

Next, I created definitions for the SW, SEE_MASK, and SE_ERR constants.

#Region " SW Constants "
Friend Enum SW As Integer
  HIDE = 0
  SHOWNORMAL = 1
  NORMAL = 1
  SHOWMINIMIZED = 2
  SHOWMAXIMIZED = 3
  MAXIMIZE = 3
  SHOWNOACTIVATE = 4
  SHOW = 5
  MINIMIZE = 6
  SHOWMINNOACTIVE = 7
  SHOWNA = 8
  RESTORE = 9
  SHOWDEFAULT = 10
  FORCEMINIMIZE = 11
  MAX = 11
End Enum
#End Region

#Region " SEE_MASK Constants "
Friend Enum SEE_MASK As Integer
  class = &H1
  CLASSKEY = &H3
  IDLIST = &H4
  INVOKEIDLIST = &HC
  ICON = &H10
  HOTKEY = &H20
  NOCLOSEPROCESS = &H40
  CONNECTNETDRV = &H80
  FLAG_DDEWAIT = &H100
  DOENVSUBST = &H200
  FLAG_NO_UI = &H400
  UNICODE = &H4000
  NO_CONSOLE = &H8000
  ASYNCOK = &H100000
  HMONITOR = &H200000
  NOZONECHECKS = &H800000
  NOQUERYCLASSSTORE = &H1000000
  WAITFORINPUTIDLE = &H2000000
  FLAG_LOG_USAGE = &H4000000
End Enum
#End Region

#Region " SE_ERR Constants "
Friend Enum SE_ERR As Integer
  SE_ERR_FNF = 2 ' file not found
  SE_ERR_PNF = 3 ' path not found
  SE_ERR_ACCESSDENIED = 5 ' access denied
  SE_ERR_OOM = 8 ' out of memory
  SE_ERR_DLLNOTFOUND = 32
  SE_ERR_SHARE = 26
  SE_ERR_ASSOCINCOMPLETE = 27
  SE_ERR_DDETIMEOUT = 28
  SE_ERR_DDEFAIL = 29
  SE_ERR_DDEBUSY = 30
  SE_ERR_NOASSOC = 31
End Enum
#End Region

Finally, I created my definition for the ShellExecuteEx function.

<DllImport("shell32.dll", CharSet:=CharSet.Auto, SetLastError:=True)> _
Friend Shared Function ShellExecuteEx(ByRef lpExecInfo As SHELLEXECUTEINFO) As Boolean
End Function

I stuck all of this into a NativeMethods class and tried the code below:

Dim info As New NativeMethods.SHELLEXECUTEINFO
info.cbSize = Marshal.SizeOf(info)
info.lpDirectory = Path.GetDirectoryName(fileToStart)
info.lpFile = Path.GetFileName(fileToStart)
info.nShow = NativeMethods.SW.SHOWDEFAULT
info.lpVerb = "open"
info.fMask = NativeMethods.SEE_MASK.FLAG_NO_UI Or NativeMethods.SEE_MASK.FLAG_DDEWAIT

If Not NativeMethods.ShellExecuteEx(info) Then
  If info.hInstApp = NativeMethods.SE_ERR.SE_ERR_NOASSOC Then
    Dim sinfo As New NativeMethods.SHELLEXECUTEINFO
    sinfo.cbSize = Marshal.SizeOf(info)
    sinfo.lpVerb = "openas"
    sinfo.lpDirectory = Path.GetDirectoryName(fileToStart)
    sinfo.lpFile = Path.GetFileName(fileToStart)
    sinfo.nShow = NativeMethods.SW.SHOWDEFAULT
    NativeMethods.ShellExecuteEx(sinfo)
  End If
End If

UPDATED (9/12/2006): Many thanks to Michael and his comment regarding using the SEE_MASK.FLAG_DDEWAIT. That fixed all of the problems I was running into regarding the above code. (see the usage on info.fMask)

A quick note about the code: Ned mentioned in his post that he got the ERROR_NO_ASSOCATION error instead of SE_ERR_NOASSOC. Well, the ERROR_NO_ASSOCATION is what is returned in the Win32 error (Marshal.LastWin32Error). The SE_ERR_NOASSOC is returned in the hInstApp (see MSDN documentation here).

See any problems with that? I certainly didn’t (and still don’t). It works like a charm for files with associations… however, it would only work one time for files without any associated program. Afterwards, it wouldn’t give me anything… no errors, nada, zilch. After a few tries, an AccessViolationException would get thrown. Why? I have no idea. I tried various things to see if I should be cleaning up memory somewhere but I couldn’t find anything. I did find out that if I just called ShellExecuteEx with the “openas” verb the first time, I wouldn’t get any problems at all. What in the world???

As a result of the strange behavior, I changed my code slightly to look like this:

Try
  Using p As New Process
    p.StartInfo.FileName = fileToStart
    p.StartInfo.UseShellExecute = True
    p.Start()
  End Using
Catch win32Ex As Win32Exception
  Dim sinfo As New NativeMethods.SHELLEXECUTEINFO
  sinfo.cbSize = Marshal.SizeOf(sinfo)
  sinfo.lpVerb = "openas"
  sinfo.lpDirectory = Path.GetDirectoryName(fileToStart)
  sinfo.lpFile = Path.GetFileName(fileToStart)
  sinfo.nShow = NativeMethods.SW.SHOWDEFAULT

  If Not NativeMethods.ShellExecuteEx(sinfo) Then
    Throw New Win32Exception
  End If
End Try

The above code is working like a charm. I still have no idea why my first example won’t work for me. If anyone has any ideas or suggestions, please let me know. I haven’t worked with Interop between managed and unmanaged code very much. My experience up to this point has primarily been an entirely managed project or an entirely unmanaged project (and that only in college).

NOTES: Here are some resources I found while researching this:

Also, be sure to look at the ShellAPI.h header file!