[Note: The below article written around 8/9/2018, was published on 10/26/2018.  I had to take a break from the below, to focus on something else, and in the meantime, the below design issue that I was able to consistently reproduce in practical ways previously, disappeared completely when I resurrected my work couple of months later.  I couldn't convince myself that the design issue does not exist, because of the issue having disappeared, so I decided to publish the below anyway, in hopes it might, at some point, start a healthy brainstorming on the topic at hand].

Intel® SGX SDK - Edger8r tool code generation and its potential constraint on trusted library design

I recently stumbled on a problem while refactoring.  I was attempting to consolidate commonly used features to a trusted static library.  The intend was to make that trusted shared library available to varied SGX projects.  I am not sure if the problem I ended up with was the result of a constraint levied by Intel® SGX SDK's Edger8r tool or an oversight on my part.  Either way, it would be good to know. 

Following is a description of the problem:

Intel® SGX SDK introduces the concept of EDL (Enclave Definition Language) files.  EDL files hold the edge routines used for interfacing between trusted and untrusted environment. At build time, Edger8r tool parses the EDL file and generates trusted or untrusted proxy/bridge code, as appropriate.  When --header-only option is specified, it generates just the equivalent header file for the EDL file.  Otherwise, it generates the proxy/bridge routine definitions as well, that eventually gets compiled into the project.  Along with the proxy/bridge definitions, relevant bookkeeping data/code as well are generated. g_ecall_table and g_dyn_entry_table objects being a couple of examples.  The former is to hold ECALLs information and the latter could possibly be a place holder for OCALLs data.  The problem is in that they are both global data which means there can't be multiple instances of it.  This further constraints which trusted library, among the many, can compile in the proxy/bridge generated code, as only one library can compile the generated definitions into the project, to avoid duplicate symbols.  This further constraints the design of rest of the trusted static libraries specific to a project.  Below example further explains the problem.

Let's assume we want to create a trusted static shared library "X" to hold common features, for use by multiple SGX projects/porting.  We would prefer for this shared library to be self-sufficient in that the relevant proxy code generated for relevant EDL files in this library be built into the library.  Then let's assume a project "P1" creates static libraries, "Y1", "Y2", "Y3", all with its own EDL files but set to generate only headers with --header-only option set.  Also, library "Y1" links to "X".  The enclave library, a dll "Z" for the project P1 links to "Y1", "Y2" and "Y3".  "Z" also, like "X", compiles in both the proxy/bridge and pertinent bookkeeping data/code generated while building an EDL without --header-only option.  This results in a scenario wherein, both "X" and "Z" end up with multiple symbols with the same name (e.g. g_ecall_table,  g_dyn_entry_table) resulting in a conflict with the linker complaining about multiple symbol definitions.  This link time error can be suppressed by using "/FORCE:Multiple" linker option but that would simply shift the problem to runtime.

To further confirm that I am not missing something obvious, I pruned through existing SGX libraries, shipped with Intel® SGX SDK, using Dumpbin, and scrutinized the EDL published for the relevant libraries.  They do not carry the symbols that are likely to eventually result in a conflict.  This would mean they were either compiled with --header-only or the EDL content didn't require generating the conflicting symbols.  In other words, the libraries they distribute, by design wouldn't have hit the issue I encountered.  Since the model I described above results in cascading dependencies requiring one or more libraries that are required to be self-contained, symbols wise, for it to be useful within multiple independent projects, we seem to have hit that design constrain.

Intel's initial vision for SGX might have confined the use of enclaves to a very narrow scope to warrant the need to consider extensive trusted shared libraries design/maintenance, the resulting dependencies and its overall impact.  Intel® SGX stands to gain from being in tune with the constraints faced by adopters of its technology, when it is used beyond its initial narrow scope.  This can only encourage further adoption of the technology, especially by classic applications hoping to capitalize on the security it provides.

As a solution to the above mentioned problem, Edger8r tool could generate more specific symbol names per trusted library and collate at runtime, proxy/bridge information from multiple libraries or use an equivalent approach so as not to shift the constraint to design level.

Intel® SGX SDK - A request to make OCALLS more seamless for DLL imports

While porting an existing application to run within an enclave environment, it is ideal to keep the changes to source to minimal.  However, existing applications do come with dependencies, the most common and obvious of which is dependency on Win32 APIs.  Projects like Microsoft Haven goes to length to shield an application's execution environment from such direct dependency on untrusted environment by leveraging Microsoft Drawbridge mechanisms like library OS and picoprocess.  However, such extensive changes are neither warranted for a significant number of applications nor feasible even, unless Drawbridge leveraged mechanisms are readily available and out of the box for seamless adoption. 

Most Intel® SGX porting efforts for existing Windows applications are likely to simply adopt Intel® SGX provided feature sets to make the necessary changes to existing source and as appropriate, for it to be functional within an enclave.  And those existing applications are likely to already rely on Win32 APIs at varied levels and thus the porting effort is likely to involve making a sizable number of OCALLs.  Currently, it takes a level of indirection to make such calls.  The source line that makes the Win32 call need to be modified to either accept the win32 API call's return value as a parameter or the function call itself need to be replaced to call to a wrapper function.

It would make for a cleaner porting if the generated code for OCALLs to DLL imports are made more seamless, where the return value from the API is still returned the classic way.  This would imply source level changes are reduced.  Simply extending the EDL file with relevant OCALLs would do.  Of course, error checking and troubleshooting might still force additional changes but with pre-existing, time tested applications that are conducive for Intel® SGX porting and likely to just work with minimal changes, it makes the porting changes look all the more simple!  For SGX related error pertaining to the OCALL, a GetLastError equivalency at Intel® SGX SDK level would work. 

The need for this relatively trivial change is unlikely to be appreciated by those who prefer Intel® SGX to be used in its intended way, with minimal and selective code running within the enclave environment.  But, this change is more useful when a significant portion of existing source is moved nearly en masse to work as an enclave application/library and that appears to be pretty much the case when porting existing libraries.

Also, this modification/addition will reduce the amount of source code changes necessary and make merging of changes to open source applications less clumsy.  From Intel's perspective, it will make Intel® SGX porting appear more effortless and even more simple!

While creating an Intel® SGX enclave, an EDL file (Enclave Definition Language file) is used to describe an enclave's trusted and untrusted functions and types.  These functions are the interface for communication between the enclave and the untrusted application.  The EDL file is read by Edger8r tool provided by the Intel® SGX SDK to generate edge/wrapper routines, which makes relevant checks and copies that ought to happen prior to and after trusted and untrusted calls.

We would like to make a request to Intel® SGX SDK team to add a minor feature to their Edger8r tool to generate prolog and epilog logging to the generated edge routines.  This would obviously be for troubleshooting convenience and in debug environment.  This request is the equivalency of what Microsoft Visual Studio compiler already provides via /Gh (Enable _penter Hook Function) compiler option.

By making it easier to troubleshoot a potentially targeted area, it is true that we might inadvertently encourage more additions to that area, thus rendering it more vulnerable.  However, those being frivolous with the interface functions are unlikely to be conservative because of lack of convenience/features of this nature.  Thus we might as well make it easy for those with a need.  After all, it is requested/recommended for development environment only.

Currently, we can workaround the lack of such a feature by simply adding another layer of indirection to EDL function calls or we could fallback on Microsoft compiler provided _penter hook for the Edger8r generated files.  The former adds an unnecessary layer of indirection while the latter is transient with the Edger8r tool regenerating the files in question for every recompilation.  An addition to Edger8r tool to add prolog/epilog hook, ought to be simple enough while helping troubleshoot problems in this area all the more easier!

Recently I encountered a BSOD on a Windows 10 Pro system.  The BSOD appear to point at an Intel audio driver.  I only took a cursory look at the problem as the problem neither appear to be in the stack we  own nor did it impede my work or repeat itself enough to warrant any more of my time.  Below is some minimal information pertaining to the crash, if it should interest relevant folks.  For access to the crash dump, please use the form here to make a request or email to info_at_kryptoguard.com

Basic crash information - 

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff80828e04096, The address that the exception occurred at
Arg3: fffff486e9b9a848, Exception Record Address
Arg4: fffff486e9b9a090, Context Record Address

Information pertaining to the module (in question) -

> lmvm IntcDAud
Browse full module list
start             end                 module name
fffff808`28dc0000 fffff808`28e88000   IntcDAud   (no symbols)          
    Loaded symbol image file: IntcDAud.sys
    Image path: \SystemRoot\system32\DRIVERS\IntcDAud.sys
    Image name: IntcDAud.sys
    Browse all global symbols  functions  data
    Timestamp:        Wed Dec  6 10:01:50 2017 (5A28065E)
    CheckSum:         000CEEC0
    ImageSize:        000C8000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Offending stack and trap frame -

!thread
THREAD ffffce8c470a2700  Cid 0004.2d9c  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0
Not impersonating
DeviceMap                 ffffa60f1ac18bc0
Owning Process            ffffce8c3bd12440       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      34652705       Ticks: 0
Context Switch Count      42             IdealProcessor: 0            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address IntcDAud (0xfffff80828e3fd50)
Stack Init fffff486e9b9ad90 Current fffff486e9b99660
Base fffff486e9b9b000 Limit fffff486e9b95000 Call 0000000000000000
Priority 15 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff486`e9b99888 fffff800`70bc6cac : 00000000`0000007e ffffffff`c0000005 fffff808`28e04096 fffff486`e9b9a848 : nt!KeBugCheckEx
fffff486`e9b99890 fffff800`70b8dc3f : 00000000`00000003 00000000`00000000 fffff486`e9b95000 fffff486`e9b9b000 : nt!PspSystemThreadStartup$filt$0+0x44
fffff486`e9b998d0 fffff800`70bb8c0d : 00000000`00000000 fffff486`e9b99a70 fffff486`e9b99f30 00000000`00000293 : nt!_C_specific_handler+0x9f
fffff486`e9b99940 fffff800`70a6fae6 : fffff486`e9b99a70 fffff486`e9b99f30 00000000`00000004 fffff486`e9b9a848 : nt!RtlpExecuteHandlerForException+0xd
fffff486`e9b99970 fffff800`70a70f03 : fffff486`e9b9a848 fffff486`e9b9a590 fffff486`e9b9a848 00000000`00000000 : nt!RtlDispatchException+0x416
fffff486`e9b9a060 fffff800`70bc0f42 : 00000000`00000000 00000000`00000000 fffff486`e9b9a7e8 00000000`00000000 : nt!KiDispatchException+0x1f3
fffff486`e9b9a710 fffff800`70bbdabf : 00000000`00000001 fffff808`28ddd00f fffff808`28ddd048 ffffce8c`41e9c1c4 : nt!KiExceptionDispatch+0xc2
fffff486`e9b9a8f0 fffff808`28e04096 : ffffce8c`41e9c100 ffffce8c`41e9c000 00000000`00000000 fffff808`28ddd00f : nt!KiPageFault+0x3ff (TrapFrame @ fffff486`e9b9a8f0)
fffff486`e9b9aa80 fffff808`28e33053 : ffffce8c`41e9c010 fffff486`00000003 ffffce8c`41e9c720 00000000`00000000 : IntcDAud+0x44096
fffff486`e9b9aab0 fffff808`28e33870 : 00000000`00000000 fffff808`00000010 ffffce8c`41e9c2b8 00000000`00000001 : IntcDAud+0x73053
fffff486`e9b9ab40 fffff808`28e19a53 : ffffce8c`41eb3ae0 00000000`00000000 00000000`00000001 00000000`00000000 : IntcDAud+0x73870
fffff486`e9b9ab90 fffff808`28e41aa3 : 00000000`00000000 00000000`00000000 00000000`00000000 fffff808`28e3f4ac : IntcDAud+0x59a53
fffff486`e9b9abf0 fffff808`28e3f96c : ffffce8c`41ea4010 00000000`00000004 00000000`00000000 ffffce8c`3bd12440 : IntcDAud+0x81aa3
fffff486`e9b9ac40 fffff808`28e3ffdb : ffffffff`fffcf2c0 ffffffff`fffcf2c0 ffffffff`fffcf2c0 ffffffff`00000000 : IntcDAud+0x7f96c
fffff486`e9b9ac70 fffff808`28e3fd9b : ffffffff`fffcf2c0 ffffce8c`41ea4010 00000000`00000080 fffff808`28e3fd50 : IntcDAud+0x7ffdb
fffff486`e9b9ace0 fffff800`70afccb7 : ffffce8c`470a2700 fffff808`28e3fd50 ffffffff`ffffffff ffffffff`ffffffff : IntcDAud+0x7fd9b
fffff486`e9b9ad10 fffff800`70bb77d6 : fffff800`7007d180 ffffce8c`470a2700 fffff800`70afcc70 ffffffff`ffffffff : nt!PspSystemThreadStartup+0x47
fffff486`e9b9ad60 00000000`00000000 : fffff486`e9b9b000 fffff486`e9b95000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

0: kd> .trap fffff486`e9b9a8f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff486e9b9aaa8 rbx=0000000000000000 rcx=fffff80828ddfc00
rdx=0000000000000003 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80828e04096 rsp=fffff486e9b9aa80 rbp=fffff486e9b9ab00
 r8=ffffe30112900180  r9=0000000000000000 r10=0000000000000004
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
IntcDAud+0x44096:
fffff808`28e04096 40387e18        cmp     byte ptr [rsi+18h],dil ds:00000000`00000018=??

Preliminary analysis (tentative/based on cursory look) - It is quite possible the content (at certain offset) of the data at address fffff808`28ddd048 passed on (via RCX register) by the third Intel frame above (IntcDAud+0x73870), which when dereferenced later (at the crashing line above) maybe the cause.

As we venture into KryptoGuard™ product and services related development work, it is not unusual for us to stumble on issues, bugs and more with the software we use towards our work.  We would like to share
with the community the result of that work or include the community in the investigation of such issues.  KryptoGuard™ developer zone was created with that intend in mind.