Mark Russinovich’s guide on the mysterious NT kernel Native API has been updated to reflected changes introduced in Windows XP and Windows Server 2003.
This API is almost entirely hidden from view, with only a handful of its function documented in generally accessible publications.
Inside The WindowsNT Native API
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
What can someone do with this knowledge?
Afterall when you program for Windows, you program at the application level using Win32.
I see this could be useful for ReactOS.
Might be helpful if you want to write something as UPX, or packer/bundler which would pack all DLL’s and EXE in one.
Also it might be useful, if you want to know how to delete your own executable. For example consider uninstaller, which had to delete itself at the end, but you can’t delete yourself if you are running – either you have to exit and give control to a batch file to delete you, and then the batch file itself (that’s possible), or register your executable for deletion on the next Windows startup.
So it might be useful.
BTW – You can install PDB symbols for most of your system32 dll’s.
Often with OpenAL I’m getting weird crashes on WaitForSingleObject, and the debugger is showing nothing (just NTKRNLDLL!!! something). If you want to get the symbols – you can download them from the Microsoft site:
http://www.microsoft.com/whdc/devtools/ddk/default.mspx
Or look for that keyword in google: xpsp1sym
Thanks for the info.
I could imagine that knowing enough of these API’s you could replace MS Win32 subsystem with your own GUI.
eg. SkyGI, or KDE/Xorg on top of the base NT OS.
Wasn’t the purpose of the Windows API a common API to the various Windows OS variants (DOS and NT based)? The DOS based existed because they were the quickest way to capture market. But now that that era is over then why not do away with the Windows API – it’s just in the way?
MS might not have much rewriting to do. I wouldn’t be surprised if MS layered products like Office, OutLook and IE skip the Windows API and goes straight to the bone. Then again, by keeping the Windows API MS layered products have the speed advantage over competitors and maybe MS thinks that they one day may introduce yet another OS.
I guess the purpose was (at first) just to be able to write software for the hardware, simple as that.
Once they realized that poking through VGA registers, setting up manually LDT/GDT to get your protected or flat-real mode is no longer fun, they realized that it seems easier if all those things are accessed through easier ways – like API.
Hell, if all computers on this world were having one and same graphics card, and the hardware was exposed – probably no one would try to make an API – but just start directly coding for it. But that’s hardly not the case, especially with the openness of x86 PC/AT machines – Vendors like Hercules totally made my world black&white even in hi-res back in the days, and it used to write it’s stuff to 0xB000 instead of 0xB800, and it did not have 320×200 VGA at 0xA000. So guess what – well you need abstraction layer to control all these devices easily.
Probably later it was used as portable API across different Windows Devices (XBOX SDK for example supports lots of Win32 functions – CreateFile, ReadFile, etc. and it’s easier to port some Win32 specific code to it).
I’ve heard lots of rumours saying that MFC was not used heavily in MS products (like you are saying Office, Visual Studio, etc.) – but I doubt they are going deeper than that. It’s good to use the Win32 on Windows. Somehow I learned to apperciate that, and that’s why I would preffer wxWidgets over anything else like Qt, GNUstep(back-win32), Tcl/Tk or anything like that trying to reinvent the controls (sometime s it’s needed, as in the case of Blender).
For example if you had installed software to translate something for you from the controls, or read it (through speech synthesizer) then it wouldn’t be possible to read that from a Qt control… But if it’s Win32 it will (You have GetWindowLong, FindFirstWindow(), etc. functions, so you can enumerate all of your windows and do something useful about it).
So it’s good (if makes sense) to use the Win32, or at least if you use wrapper GUI/OS library to go through it, and just emulate as little as possible.
Claus, Microsoft cannot do away with the Win32 API. Too much of their software partners are dependant on it as we speak. Also think about all the legacy applications out there, that expect a Win32 subsystem to be present.
If MS dropped Win32 in the next Windows release and presented a new API based on the Kernel API, you’d get a revolt. It would mean that all existing Windows software would have no place to go but older versions of Windows. The latest and greatest from MS would come into the world without an installed software base.
It would mean suicide for MS. The new Windows would not be able to run all previous software. End-users would be outraged about having to purchase all their already purchased applications again. Third party software developers would be infuriated about having to port their existing flagship products to the new Win API.
The most shure way MS has to push people to other OSes is to drop Win32 like a brick. As MS is not mentally certifiable, they will introduce a new API very careful and above all veeerrrryyy slooooow.
Dealing with the native API is needed for realizing on-access antivirus scanning of files, for example. To realize that successfully across the various Windows-es, one needs IFS Kit which is 800+ USD without tax
Considering this is needed mostly for antivirus vendors, this is cheap; however, it is obviously simply a way for Microsoft to generate additional revenue out of nothing — they could very well put this kit in the otherMSDN documentation, for which you actually pay quite enough (and some parts of the world pay substantially more than that).
Basiclly, application developers now have a method to manually acquire virtual memory, manipulate process entries, suspend other threads (bypassing security) etc. By accessing the kernel API, you can bypass Windows file and process protection, and cause all sorts of strife. Not to mention direct hardware access. A nasty bit of spyware now has the ability to directly modify protected Windows system files / resources.
I think the world was better of while the API was hidden. Now we’re just counting the days before something really sinister strikes …
LOL
…
that was a joke, wasn’t it?
>>I think the world was better of while the API was hidden.
>>Now we’re just counting the days before something really
>>sinister strikes …
It is not necessary to go ‘that’ deeper to do something sinister. And believe me the people who are able to do sinister things at that level already got all they need.
As the article states these APIs do not bypass the security model. The Windows security model is based in the kernel and not in the Win32 subsystem. Any calls you make to these APIs will still take place in the context of the running process with that processes security token and permission set.
You basically only need to use these apis for low level tools like virus scanners and file system defragmenters, tools that need to operate before the Win32 subsystem has been initialised.
It is not necessary to go ‘that’ deeper to do something sinister. And believe me the people who are able to do sinister things at that level already got all they need.
Yes, nut now a whole new bunch of incompetent idiots will be able to go that deep for the first time thanks to the documentation.
Basiclly, application developers now have a method to manually acquire virtual memory, manipulate process entries, suspend other threads (bypassing security) etc. By accessing the kernel API, you can bypass Windows file and process
So you want security by obscurity ? Doesn’t work well.
At any rate, the kernel _OFCOURSE_ does the appropriate
security checks. Please don’t think you can bypass security
with this. (And if you can, consider it a bug that hopefully
will be fixed.) As the article discuss, these are the equivialent of *nix syscalls, just that on unix they are
public apis.
Please engage your brain a bit more.
What can someone do with this knowledge?
Read Inside Windows NT: http://www.amazon.com/exec/obidos/tg/detail/-/1572316772/qid=110190…
One of NT’s original design requirements was compatibility between a diverse set of APIs: POSIX and OS/2 being the original set, with Win32 being introduced later. These APIs are very different — compare fork(2) to CreateProcess(), and the file model (POSIX lets you delete open files, Windows doesn’t, as others have mentioned).
NT’s solution? Have a microkernel which supports the implementation of the union of all the APIs, with separate environment subsystems to implement the actual APIs. This is how Windows NT 3.0 supported Win32, POSIX, and 16-bit OS/2.
16-bit OS/2 support has been effectively dropped, but POSIX support has been revamped as Windows Services for Unix: http://www.microsoft.com/windows/sfu/default.asp
So, what can someone do with this knowledge? Create new API sub-systems. Imagine a purely .NET managed-code subsystem, with no Win32 support. Sure, you’d lose compatibility, but you’d also lose any potential buffer overflows present in the native Win32 code, which would be good for security. Or a BeOS subsystem, or….
>So, what can someone do with this knowledge? Create new API >sub-systems. Imagine a purely .NET managed-code subsystem, >with no Win32 support. Sure, you’d lose compatibility, but >you’d also lose any potential buffer overflows present in the >native Win32 code, which would be good for security. Or a >BeOS subsystem, or….
Until you have managed code in the kernel you are still going to have security risks. Drivers can be hacked just like any other software.
To all those “secret API” conspiracy theorists, this has been public knowledge for years. It’s the same API that’s always been used in NT4 for the “…..” phase (blue screen, white text and all). It’s still used in 2000 and XP but the blue screen text output has been diverted to a more graphical screen (as in the chkdsk /f c: screen)
You’ve not stumbled across any secrets, programs taking advantage of this API, and relevent documentation, have been around for about a decade.
It looks like that “API” are just syscalls to me. And win32 is the equivalent of the GNU C library (ie: a posix implementation).
That’s what I like about linux, things like that are not hidden but open for everybody which is great.
Of course there’s no “a huge security risk” with that. What makes people think that? I bet some columnists of online newspapers will read that and publish news about a “huge security hole in Windows”. It wouldn’t be the first time…
First of all the pointed resource is not the best for native APIs of windows. You should look here
http://undocumented.ntinternals.net/
or
the book “Windows NT/2000 Native API reference”. This book is really worth having.
A word about Native API. Native APIs are simply system calls in windows. They are the functions provided by ntoskrnl or NT kernel as to say. There is a good reason to hide them because as they are part of kernel, their interface tends to change in future though not necessary that it will change. Microsoft doesn’t make public because then it will be morally bound to not change those interfaces.
Microsoft tries to provide all the functionality to access windows kernel in Win32 APIs. Sometime this functionality is not sufficient and people have to use Native APIs to get their task done. What they should do along with it is, raise voice in MS forums to get the required APIs added in Win32. This way once an API is added in Win32, its compatibility is assured.
These APIs are not a security hole at all. Someone got to be really stupid to think that these can cause security problem, because all Win32 functions use these native APIs and thus if knowledge of Native APIs could make windows less secure then so does knowledge of Win32. Hence NO they don’t pose a security problem or information disclosure threat at all.
…I’m lost for a way to monitor more than 64 event objects simultaneously using a single thread…
Worst arbitrary limit ever
why only one thread? that’s unix mentality. Use completion ports and a pool of threads. There’s even an article on completion ports from the same guy that wrote this one: http://www.sysinternals.com/ntw2k/info/comport.shtml
Use completion ports and a pool of threads.
Completion ports aren’t a feasible solution for a majority of applications which monitor for socket readyness [e.g. WSAEventSelect()] rather than using asynchronous I/O. Writing an appliation portable between Windows and a variety of POSIX systems is virtually impossible, and a clean abstraction is rather difficult.
Furthermore, use of threads will substantially increase application complexity and difficulty of debugging Completion ports are also a more server-oriented feature, and applications which want to monitor a variety of other events in addition to sockets are essentially screwed when completion ports are used, as an additional thread is needed to multiplex the event obects while an entire thread pool is needed to process completion ports.
Completion ports work if your application uses a completely asynchronous I/O-based model. Otherwise you’re screwed.
I never used Completion Ports, but needed to make my ReadFileEx (asynchronous) to callback a specific function once it’s ready. The problem is that to do that I had to manually call SleepEx(1) once in a while, and that’s not really what I want – I want without it.
Would completion ports help me? I never used them, and have bit trouble understanding them.
If you are a sys admin, like me, and you are trying to debug a memory or user dump, knowing the API calls can be very useful in reading the stack in windows debugger.
well, windows i/o model is asynchronous and unix/posix is synchronous, and of course the nt kernel is of vms descent and completely different from any unix. So for writting portable apps you either use java, mono, whatever, or when coding your app abstract the i/o part of it (into several classes for example), and then have a different i/o layer/classes for each operating system.
In any case you can still use completion ports with sockets, and most unixes also provide some sort of asynchronous i/o. Solaris has “completion events” and freebsd has “kqueue”, linux asynchronous i/o support is kind of bad though.
http://developers.sun.com/solaris/articles/event_completion.html
Also, you could use fibers to monitor sockets instead of threads: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dl…
“It looks like that “API” are just syscalls to me. And win32 is the equivalent of the GNU C library (ie: a posix implementation).”
Yes somewhat like that.
“That’s what I like about linux, things like that are not hidden but open for everybody which is great.”
Not really, in Linux thay sometimes removes syscalls, if one uses one of those syscalls the code must be rewritten. The NT kernel is designed so that the internal API can be changed while preserving the WIN32 calls unchanged, it’s a matter of design. The only reason thay are not publicized are that they can (and have) be changed/removed between different kernels.
“Of course there’s no “a huge security risk” with that. What makes people think that? I bet some columnists of online newspapers will read that and publish news about a “huge security hole in Windows”. It wouldn’t be the first time…”
What? The internal syscalls are secure and don’t contain any backdoors or something.