This is a fundamental problem with VCL styles and the way that they style menus. The styling is implemented with a process wide hook. Specifically a CBT hook installed by a call to SetWindowsHookEx from TCustomStyleEngine.CreateSysHook in the Vcl.Themes unit. In fact, the hook applies just to the GUI thread, but that is process wide in the sense that there is exactly one GUI thread in the process.
Since you have multiple instances of the VCL in your application (one in the DLL and one in the application), two hooks are installed. That is one too many. The hook installed most recently (the DLL as it happens) wins, and that's why the DLL menu styling infects your executable. And why you encounter an access violation. The DLL is trying to operate on a menu that belongs to the executable. And so, in spite of your best efforts, you've ended up with the DLL code accessing VCL objects from the host executable.
There's no simple way to work around this and support styles fully in both modules. What we have here is a fundamental consequence of the design. The system was not designed to support multiple VCL instances. If you wish to use VCL styles in multiple modules, then the designers expect you to use runtime packages.
I suppose that you might be able to get some traction by operating the DLL out of a completely different thread. That would involve loading the DLL from that different thread so that the VCL is initialized in the thread. And all calls to the DLL would have to be from that thread. And you'd need to run a message loop in that thread. It's possible that you might be able to make that work, but I doubt it. Even with all the provisos mentioned you still have to handle the fact that you have two GUI threads which presents all sorts of issues with the input queue handling.
Perhaps another approach would be to uninstall the hook from the DLL. So long as your DLL is not showing menus then you may well be able to get away with uninstalling that hook. It would disable styling for menus shown by the DLL, but perhaps that's acceptable.
This version of your DLL (after I simplified it somewhat also) uninstalls the hook.
library VCLStyleDLL;
{$R 'Style.res' 'Style.rc'}
uses
VCL.Styles,
VCL.Themes,
VCL.SysStyles; // to gain access to TSysPopupStyleHook
{$R *.res}
begin
TStyleManager.TrySetStyle('Glossy', false);
TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook);
end.
With this version of the DLL, the host executable does not suffer the problems your describe in your question.
Best Answer
This is known issue with VCL Styles http://qc.embarcadero.com/wc/qcmain.aspx?d=106224
Also see this issue in Embarcadero's newer QC site: https://quality.embarcadero.com/browse/RSP-11572 --- it's been 3 years since initially reported, and still not fixed. If enough people vote for that issue, maybe it will get some attention.
As workaround you can load proper 16x16 icon into form's
Iconproperty.In order for that to work you have to also set
Application.MainFormOnTaskBar := false;in your.dprfileHowever that has some other undesirable effects because it will disable Windows Vista or Windows 7 Aero effects, including live taskbar thumbnails, Dynamic Windows, Windows Flip, and Windows Flip 3D. See: MainFormOnTaskBar
In any case do not change your application icon size because it is the worst solution.