Earth Cubed

Distributed Climate Science and Computing

API/Object Viewers/Memory Mapping/

The more code a programmer can reuse the more efficient they can be. In windows this could mean reusing com/ole components and other APIs. Here are two useful programs for viewing APIs:

OLE/COM Object Explorer 1.1
http://www.softpedia.com/progDownload/OLE-COM-Object-Explorer-Download-42531.html

Windows API Viewer
http://www.activevb.de/rubriken/apiviewer/index-apiviewer.html

I was inquiring about how to manage the transfer of large amounts of data between programs and I was pointed to two interesting concepts:

All modern operating systems include a facility called “memory mapping,” which maps a range of addresses in the program’s virtual address space to a file. If you read from those addresses, you’ll get data from the file. It is up to the operating system to determine whether to load the data into RAM all at once, or to read it from the disk in chunks as necessary.
…….
If you’re trying to share large amounts of memory between two programs running on the same computer, you should note that all modern operating systems provide mechanisms for shared memory. These shared memory segments can be mapped into the virtual address space of multiple programs simultaneously. Two or more programs can read or write to the shared memory exactly as if it were normal, private memory. (But you should include some thread-safety mechanisms, like mutexes, to make sure your programs won’t step on each other’s toes.)

If you’re trying to share large amounts of memory between programs running on separate computers, use MPI or some other multi-processing library.

http://www.physicsforums.com/showthread.php?t=333182

Here is what wikipedia has to say about memory maps:

The primary benefit of memory mapping a file is increased I/O performance, especially when used on small files. Accessing memory mapped files is faster than using direct read and write operations for two reasons. Firstly, a system call is orders of magnitude slower than a simple change of program’s local memory. Secondly, in most operating systems the memory region mapped actually is the kernel’s file cache, meaning that no copies need to be created in user space. Using system calls would inevitably involve the time consuming operation of memory copying.

Certain application level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in flat file data storage or configuration files) requires disk access only when a new page boundary is crossed, and can write larger sections of the file to disk in a single operation.

A possible benefit of memory-mapped files is a “lazy loading”, thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but the system only needs to load the smaller page-sized sections as data is being edited, similarly to demand paging scheme used for programs.

http://en.wikipedia.org/wiki/Memory-mapped_file#Benefits

The windows utility to do this is called CreateFileMapping
http://msdn2.microsoft.com/en-us/library/aa366537.aspx

As for multi-processing library the following was recommended:
http://scv.bu.edu/documentation/tutorials/MPI/MPI_text.html

I haven’t found much but the following wikipedia link seems relevant:
http://en.wikipedia.org/wiki/Cluster_(computing)

Advertisements

August 29, 2009 - Posted by | Computer Science and Modeling

2 Comments »

  1. As a side note I descovered OLE/COM Object Explorer 1.1 because I was thought it might be cool to represent matrices in excel using Microsoft equation editor:
    http://www.physicsforums.com/showthread.php?t=333227

    I was having trouble finding what dll was used by Microsoft equation editor. However, I think Microsoft equation editor uses OLE instead of com. I thought they were the same but when I looked at it using the object explore what I saw for the relevant dll was Ole32.dll which I think is more of a general utility then would be used in a com interface.

    Comment by s243a | August 29, 2009 | Reply


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: