THE HACKER'S VIEW OF THE COMMODORE 64

Jun 1, 2015

ICU64 for Frodo Redpill v0.1.6 - Released

The creation of memory maps gets easier with this release.

New features in the memory view:

BASIC program layout
Load or type a BASIC program and hit ctrl-L in the memory view to watch the memory map of its code. After RUN, you have to press ctrl-L periodically to update that map, because a running BASIC program updates the layout of the data area (which is located after the code area) as it meets new variables for the first time. Enable the text view (ctrl-A) to get a more meaningful view (or type LIST in the emulator but this is like cheating!)

Data Classification (Order In Chaos)
The data classification view colorize the memory in a chaotic still consistent way. And using only your pattern recognition abilities you can organize the data even if you don't understand their meaning!
So far, watching the internals of the C64 in action may was something, yet you may have this feeling: "wow! I see everything :) but still I understand almost nothing :(". Well, the new feature lets the CPU to interpret  to us the meaning of the data using a color language. And no, you don't have to learn another color code, since these colors are meaningless this time (it's just a meaningless 'alphabet'). What's important is their patterns (i.e. the 'words' and 'phrases' that they form). These patterns are not the usual data patterns (as in the grayscale view), they are event patterns arranged in space that highlight the internal structure of the data as being used by the code (who always knows their meaning, and this is where we rely).
More particular, this visualization is based on 'what code access what data', and actually it's a simple hash function of the "last access" address that located in the upper left corner of every cell (visible in deep zoom). So, actually, this information was always there in a numerical/textual form, proper only for serial/boring reading. By translating this information into colors, patterns emerge (from chaos!) that can be recognized immediately by the human brain in parallel way, and organized manually. This view is especially useful to find large internal structures like arrays or lists of data, and analyze their entries down to their most individual parts.

Map Editing
Some new functions that will help you with your memory maps:
    Enter / Shift-Enter                  : Split / Join line
    Backspace / Shift-Backspace    : Move line Left / Right

(the above operations never overwrite non empty cells)
    Delete / Shift-Delete                : Delete cell / line
    right click / Insert*                  : Pick cell / Put cell++ 

    F2*                                       : Rename cell (change address)
Note that the editing functions are minimal yet, and they are line oriented. If you want to move blocks around you must export the map and use some spreadsheet application.
 * edit: bugs found on these operations that may mess your map, so avoid them (fixed in version 0.1.7)

Drag 'n' Drop
If you drop a .txt file in the memory view, it will open as a memory map. Any other file will be imported as a .prg file (i.e. will be loaded in the RAM at the location that the first two bytes specify). BTW, the Frodo Redpill window accepts .fss files (Frodo snapshots).

Press F1 on every window (even in the main menu) to see more functions.

You can also download some generic memory maps that may help you. Especially the RAM only map, it's a good start point to create the map of a game.


How to create a memory map, in practice:
- start a game in the emulator and wait until it gets into its main loop
- from the memory view of icu64, import (ctrl-i or drag'n'drop) the RAM only map (so you don't mess with the ROM and the MMIO while editing the map)
- press shift-ctrl-X to clear the colors and thus start a new log (do this at any time)
- (optional) switch to view '1' or '3' (cpu events) and play with the game for a while until you locate its code areas to remove them (since the data classification is meaningless with the code parts). Use 'shift-delete' to delete entire lines.
- switch to view '2' (vic events) to find the bitmaps, text screens, charsets, and sprites, all of which have 'standard' layouts usually (the graphics view will help you with this).
- switch to view '4' (data classification) and play with the game as much as possible, so the code to exploit  much (if not all) of its data. As you start seeing similar patterns in the memory view,  hit 'enter' over the beginning of each pattern to send it to a new line. The goal is to align the similar patterns vertically. To join two lines, hit 'shift-enter' on the first line.
- if some colors are very similar, hit '0' to get a different set of colors
- note that even with an incomplete map, you can probably locate lists of pointers/offsets (they have very characteristic patterns) to the patterns you have found so far, and this way you can find ALL the addresses of the entries in a list/table/etc.
- to tidy up the map export it (ctrl-O) as a text file, open it with Notepad, and copy everything or a part. Start a new sheet on Excel, select all of its cells and mark them as 'Text' (ctrl-A > right click > Format cells > Text). Paste the map to the sheet and do your edits (e.g. move blocks around). Finally, copy & paste everything  back to Notepad, save the map, and import it again to icu64. (edit: in v0.1.7 you can move blocks around, so there is no need for external tools)
- enjoy the new view!

30 comments:

  1. I just noticed this update today, I haven't tried it just yet, however, judging by the description this looks great! This should really help in reverse engineering: visualization is based on 'what code access what data'. I suppose by neat your map you mean that a spreadsheet can easily align row and columns. Unfortunately my anti-virus doesn't seem to like frodorp.exe for some unknown reason, nor did it like previous versions of frodorp.exe. Any chance of a WinVice version, or is only a Frodo version planned? Are there any videos demonstrating the new features?

    ReplyDelete
    Replies
    1. If you don't trust it, run it on vmware or similar (i suppose that i can't convince you that it's clear, anyway). Data classification can't supported in the current vice versions, only in WinVice Redpill (whenever will be ready). And no video this time, only text (the last paragraph is your guide). Just don't afraid the chaos... you are supposed to put the order!

      Delete
  2. I just got a reply from the anti-virus company and after analyzing frodorp.exe and icu64.exe they agree that it was a false positive and will place them on their whitelist which will taken affect within a day. I've already begun memory mapping a few C64 games, I hope that with these new tools the mapping process will be much faster & easier!

    ReplyDelete
  3. The SID monitor is really cool! Is it possible to disable certain voices by chance? I assume the 1st number is frequency, but I'm not sure about the 2nd through 4th numbers. Also I'm not sure what is on the far right on the window.

    ReplyDelete
    Replies
    1. No, it's not possible.
      The SID view has the following layout:

      [first three columns]
      frequency (0..ffff)
      pulse width (0..fff)
      control register (8 bits)
      a d s r (4 hex digits)

      adsr graph (rough approximation)

      [last column]
      V filter freq (0..7ff)
      O FX + filter enable (4 bits of d417 & 0x0f)
      L V3 + filter mode (4 bits of d018 & 0xf0)
      U resonance (1 hex digit of d417 & f0)
      M
      E

      Delete
    2. Thanks for the info! I found a good article about the SID chip in the magazine Compute! issue 49 http://www.atarimagazines.com/compute/issue49/424_1_Programming_64_Sound.php

      Delete
  4. Doing ctrl-A (after doing ctrl-L) shows the wrong PETSCII characters http://sta.c64.org/cbm64pet.html

    Such as 0x53 shows a heart when it should show a capital S. Also the first 4 bytes of each line should never interpreted as PETSCII, but I realize this is an early version.

    ReplyDelete
    Replies
    1. ctrl-a shows the correct PETSCII chars. BASIC is using a different -internal- encoding, for technical reasons. After 'ctrl-a', hit 'a' once, to get the lower-case charset which is slightly better for BASIC programs (hit shift-a or ctrl-shift-a to return to the upper case charset).

      The memory view is a general view. Hitting ctrl-L, icu64 creates automatically a map that shows you only how a BASIC program is encoded in memory. The layout of its bytes. A full BASIC lister will be a special view in the future.

      Delete
    2. Thanks, I didn't realize one could change the case. So 'a' goes forward, 'shift-a' goes backward, and 'ctrl-shift-a' goes back to default. I noticed that you have to press 'a' 34 times to return to where you started. Is there a way to know the memory address of the charset it is using?

      Delete
    3. Ignoring the first two ROM charsets, multiply the times (starting from zero) you hit 'a' with 2048 (= 65536/32). Or find the charset in the graphics view, press F10, and see the address of the memory under the mouse pointer on the status bar.

      Delete
    4. Thanks for the info. I assume the first two ROM charsets can only be read by the VIC chip? Are the remaining 32 charsets RAM or are some of those ROM also?

      Delete
    5. The CPU has full access everywhere except the first two bytes of the RAM. The VIC has -read only- access to the CHARROM, the COLORRAM, and the RAM (except the ranges 1000-1fff and 9000-9fff, where sees two copies of the charrom). Thus the VIC can see actually only 2 charsets in the ROM + 28 charsets in the RAM.
      Google for the 'vic article', the best article about how the VIC works. You will learn lots of details that can't described in comments.

      Delete
  5. Thanks to ICU64 I just discovered one method programmers used to prevent listing a BASIC program: quite simply put character 0xCC after a REM token 0x8F which crashes the listing routine! Easily fixed, just replace 0xCC with a character such as capital S 0x41.

    ReplyDelete
  6. Also thanks to ICU64 I just discovered every detail on how BASIC multi dimensional arrays are stored in RAM! I was thinking perhaps elements could be separated by 1 spacing, and dimensions separated by 2 spacings?

    ReplyDelete
    Replies
    1. BASIC arrays will be visualized as tables of tables in the future, so their elements to be clear.

      Delete
  7. I like the new mnemonic labels with addressing modes, it makes it much easier to follow the code, no more looking up opcodes! Cyan for opcode and green for operands. Maybe BASIC tokens could be labeled like this as well. I assume '..' for zero page addressing and '....' for non-zero page addressing.

    I noticed that it only labels code that has been executed which makes sense since that is obviously code. Though latent code can go unlabeled. For example in M.U.L.E. the BNE at 72A5 always branches during the starting title screen, but the code that follows, 72A7, hasn't executed yet while waiting and the thus not labeled yet, though I can understand it would be a hassle to have it pursue all of the code branches.

    ReplyDelete
    Replies
    1. Colorizing a BASIC program needs a slightly different data classification schema, and is planned.

      Blind disassembly is not accurate, because it depends from the address you start the disassembly. icu64 visualize only what happens, to remain accurate and minimal.
      You can hit F10 on the emulator to start the embedded debugger and disassemble (d xxxx) any code not executed yet.

      Delete
    2. I look forward to the BASIC program colorization! Maybe the tokens can be labeled like the opcodes are now, with orange text at the bottom of the cell. I'd imagine that determining tokens would be much more difficult than determining opcodes as opcodes have a fixed number of bytes for the operand, unlike with tokens.

      Regarding opcodes: perhaps when you click on a cell there can be a option to treat the cell as an opcode for these latent code sections.

      Delete
  8. What's bugging me the most is frodorp.exe crashing. Easily reproducible by clicking the title bar, less than 20 clicks or so and it crashes. I noticed that clicking on the title bar freezes the program, so what I'm guessing is that if you freeze it when it's executing a certain routine it crashes and it's a matter of bad luck if that's when it is clicked. I'm wondering why it freezes at all, why not just not paint while title bar is clicked, but continue to execute the program.

    ReplyDelete
    Replies
    1. Never crashed to me. What Windows you are using and what is the message of the crash?
      BTW, it's normal for all Windows applications to freeze when you click the title bar. And even I can disable this behavior, I think it's a handy way to pause the emulator for a short time. VICE has similar behavior, while CCS64 don't.

      Delete
    2. Ya clicking on the title bar does seem to pause other Windows apps as well, I guess I didn't pay attention to it before. Selecting frodo from the taskbar to put into focus avoids this problem. I noticed that even when you first start frodorp.exe and you click on the title bar it makes crackling noises, which is odd. This is with Windows 7, I should be upgrading to Windows 10 soon.

      Delete
  9. What I'm currently resorting to when labeling memory map sections (copy paste the following to a text editor with fixed width characters, such as notepad):
    0000 0000 0000 ---- 0000 0000 0000
    0000 ---- ---- ---- ---- 0000 ----
    ---- 0000 ---- ---- ---- 0000 ----
    ---- ---- 0000 ---- ---- 0000 ----
    0000 0000 0000 ---- ---- 0000 ----
    In a text editor (and in the memory map) this says 'ST'. So what should take only say 2-4 bytes ends up taking 70 or so bytes. It would be nice if letters could take the place of memory address cells for comments.

    ReplyDelete
  10. GREAT RELEASE!!!

    It saved me some hours of debugging!
    If you can port it to Vice this will be awesome!

    /Fix

    ReplyDelete
  11. Any chance you're going to open source this - maybe a github repository? Great work, the visualization is awesome.

    ReplyDelete
    Replies
    1. Not any soon for the icu64. But the redpill versions of the emulators (frodo and vice) will be distributed with their source code when v0.2 will be released.

      Delete
  12. Hej Mathfigure! The new features look incredible! Could you possibly record a short video demoing all of them? Thanks!

    ReplyDelete
    Replies
    1. Instead, read the paragraph "How to create a memory map, in practice", because describes what such a video could show. I'll do a video when v0.2 will be available (still unknown when)

      Delete