Ray tracing in Notepad.exe at 30 frames per second

A few months ago, a post was posted on Reddit describing a game that used an open source Notepad clone to handle all input and rendering. Reading about this, I thought it would be great to see something similar working with the standard Windows Notepad. Then I had too much free time.





I ended up creating a Snake game and a small ray tracer that use standard Notepad for all input and rendering tasks, and along the way learned about DLL Injection, API Hooking, and Memory Scanning. Describing everything I have learned in the process may be interesting reading for you.





First I want to talk about how memory scanners work and how I used them to turn notepad.exe into a render target at 30+ frames per second. I'll also talk about a ray tracer I built for rendering in Notepad.





Sending key events to notepad

I'll start by talking about dispatching key events to a running Notepad instance. This was the boring part of the project, so I'll be brief.





Win32 (, ), , , , , «», , . , Visual Studio Spy++, , .





Spy ++ Notepad
Spy++

Spy++ , , , «». , , Win32, HWND , . HWND :





HWND GetWindowForProcessAndClassName(DWORD pid, const char* className)
{
  HWND curWnd = GetTopWindow(0); //0 arg means to get the window at the top of the Z order
  char classNameBuf[256];

  while (curWnd != NULL){
    DWORD curPid;
    DWORD dwThreadId = GetWindowThreadProcessId(curWnd, &curPid);

    if (curPid == pid){
      GetClassName(curWnd, classNameBuf, 256);
      if (strcmp(className, classNameBuf) == 0) return curWnd;

      HWND childWindow = FindWindowEx(curWnd, NULL, className, NULL);
      if (childWindow != NULL) return childWindow;
    }
    curWnd = GetNextWindow(curWnd, GW_HWNDNEXT);
  }
  return NULL;
}
      
      



HWND , PostMessage WM_CHAR.





, Spy++, 64- . , Visual Studio 2019 . Visual Studio «spyxx_amd64.exe».





, 10 , , , , 30 . , .





CheatEngine

CheatEngine. , . , // , . .





CheatEngine , . , . , :





  • , (, 100)





  • - , (, 92)





  • , ( 100), , 92





  • , (, , , )









CheatEngine and Notepad "made friends"
CheatEngine ""

, ,   , , . CheatEngine, ( ) . :





  1. UTF-16, , UTF-8.





  2. , CheatEngine (, ?)





  3. . ,





, , .





, , . CheatEngine, - , :





FOR EACH block of memory allocated by our target process
    IF that block is committed and read/write enabled
        Scan the contents of that block for our byte pattern
        IF WE FIND IT
            return that address

      
      



~ 40 .





, , — .





64- Windows ( 0x00000000000 0x7FFFFFFFFFFF), 0 VirtualQueryEx .





VirtualQueryEx MEMORY_BASIC_INFORMATION



, , , VirtualQueryEx , . MEMORY_BASIC_INFORMATION



.





MEMORY_BASIC_INFORMATION



, BaseAddress RegionSize VirtualQueryEx





char* FindBytePatternInProcessMemory(HANDLE process, const char* pattern, size_t patternLen)
{
  char* basePtr = (char*)0x0;

  MEMORY_BASIC_INFORMATION memInfo;

  while (VirtualQueryEx(process, (void*)basePtr, &memInfo, sizeof(MEMORY_BASIC_INFORMATION)))
  {
    const DWORD mem_commit = 0x1000;
    const DWORD page_readwrite = 0x04;
    if (memInfo.State == mem_commit && memInfo.Protect == page_readwrite)
    {
      // search this memory for our pattern
    }

    basePtr = (char*)memInfo.BaseAddress + memInfo.RegionSize;
  }
}
      
      



, / .State .Protect. MEMORY_BASIC_INFORMATION



, , , 0x1000 (MEM_COMMIT



) 0x04 (PAGE_READWRITE



).





( , , ). . ReadProcessMemory.





, , . , , . , .





char* FindPattern(char* src, size_t srcLen, const char* pattern, size_t patternLen)
{
  char* cur = src;
  size_t curPos = 0;

  while (curPos < srcLen){
    if (memcmp(cur, pattern, patternLen) == 0){
      return cur;
    }

    curPos++;
    cur = &src[curPos];
  }
  return nullptr;
}
      
      



FindPattern() , . , FindPattern, , . .





char* FindBytePatternInProcessMemory(HANDLE process, const char* pattern, size_t patternLen)
{
  MEMORY_BASIC_INFORMATION memInfo;
  char* basePtr = (char*)0x0;

  while (VirtualQueryEx(process, (void*)basePtr, &memInfo, sizeof(MEMORY_BASIC_INFORMATION))){
    const DWORD mem_commit = 0x1000;
    const DWORD page_readwrite = 0x04;
    if (memInfo.State == mem_commit && memInfo.Protect == page_readwrite){
      char* remoteMemRegionPtr = (char*)memInfo.BaseAddress;
      char* localCopyContents = (char*)malloc(memInfo.RegionSize);

      SIZE_T bytesRead = 0;
      if (ReadProcessMemory(process, memInfo.BaseAddress, localCopyContents, memInfo.RegionSize, &bytesRead)){
        char* match = FindPattern(localCopyContents, memInfo.RegionSize, pattern, patternLen);

        if (match){
          uint64_t diff = (uint64_t)match - (uint64_t)(localCopyContents);
          char* processPtr = remoteMemRegionPtr + diff;
          return processPtr;
        }
      }
      free(localCopyContents);
    }
    basePtr = (char*)memInfo.BaseAddress + memInfo.RegionSize;
  }
}
      
      



, , «MemoryScanner» github. ! ( , ymmv, ).





UTF-16

, UTF-16, , FindBytePatternInMemory (), UTF-16. . MemoryScanner github :





//convert input string to UTF16 (hackily)
const size_t patternLen = strlen(argv[2]);
char* pattern = new char[patternLen*2];
for (int i = 0; i < patternLen; ++i){
  pattern[i*2] = argv[2][i];
  pattern[i*2 + 1] = 0x0;
}
      
      



, , WriteProcessMemory . , , , Edit.





, Win32 api InvalidateRect, .





, :





void UpdateText(HINSTANCE process, HWND editWindow, char* notepadTextBuffer, char* replacementTextBuffer, int len)
{
  size_t written = 0;
  WriteProcessMemory(process, notepadTextBuffer, replacementTextBuffer, len, &written);

  RECT r;
  GetClientRect(editWindow, &r);
  InvalidateRect(editWindow, &r, false);
}
      
      



. , , , , ,  .





:

















. MoveWindow , , .





, , ( ) , . MoveWindow , WM_CHAR . , .





, , , WM_CHAR.





, . github , .





void PreallocateTextBuffer(DWORD processId)
{
  HWND editWindow = GetWindowForProcessAndClassName(processId, "Edit");

  // it takes 131 * 30 chars to fill a 1365x768 window with Consolas (size 11) chars
  MoveWindow(instance.topWindow, 100, 100, 1365, 768, true); 

  size_t charCount = 131 * 30;
  size_t utf16BufferSize = charCount * 2;

  char* frameBuffer = (char*)malloc(utf16BufferSize);
  for (int i = 0; i < charCount; i++){
    char v = 0x41 + (rand() % 26);
    PostMessage(editWindow, WM_CHAR, v, 0);
    frameBuffer[i * 2] = v;
    frameBuffer[i * 2 + 1] = 0x00;
  }

  Sleep(5000); //wait for input messages to finish processing...it's slow. 
  //Now use the frameBuffer as the unique byte pattern to search for
}
      
      



, , , .





. , (Consolas, 11pt), - WM_SETFONT , , . Consolas 11pt , .





, , , . , ScratchAPixel . , .





, . WriteProcessMemory ( , ), , ( * 2 (- UTF16)). , WriteProcessMemory . :





void drawChar(int x, int y, char c); //local buffer
void clearScreen(); // local buffer
void swapBuffersAndRedraw(); // pushes changes and refreshes screen. 
      
      



, , (131 x 30), , «» . , , , , ascii. , .





. , , . , «» , , .





float aspect = (0.5f * SCREEN_CHARS_WIDE) / float(SCREEN_CHARS_TALL);
      
      



, , , . , , , WM_VSCROLL, « » , . , , , , .





2: Boogaloo!





The next (and last) part of my quest to create a real-time game in Notepad was figuring out how to handle user input. If you want more, the next post can be found here !








All Articles