API/Object Viewers/Memory Mapping/

The more code a programmer can reuse the more efficient they can be. In windows this could mean reusing com/ole components and other APIs. Here are two useful programs for viewing APIs:

OLE/COM Object Explorer 1.1
http://www.softpedia.com/progDownload/OLE-COM-Object-Explorer-Download-42531.html

Windows API Viewer
http://www.activevb.de/rubriken/apiviewer/index-apiviewer.html

I was inquiring about how to manage the transfer of large amounts of data between programs and I was pointed to two interesting concepts:

All modern operating systems include a facility called “memory mapping,” which maps a range of addresses in the program’s virtual address space to a file. If you read from those addresses, you’ll get data from the file. It is up to the operating system to determine whether to load the data into RAM all at once, or to read it from the disk in chunks as necessary.
…….
If you’re trying to share large amounts of memory between two programs running on the same computer, you should note that all modern operating systems provide mechanisms for shared memory. These shared memory segments can be mapped into the virtual address space of multiple programs simultaneously. Two or more programs can read or write to the shared memory exactly as if it were normal, private memory. (But you should include some thread-safety mechanisms, like mutexes, to make sure your programs won’t step on each other’s toes.)

If you’re trying to share large amounts of memory between programs running on separate computers, use MPI or some other multi-processing library.

http://www.physicsforums.com/showthread.php?t=333182

Here is what wikipedia has to say about memory maps:

The primary benefit of memory mapping a file is increased I/O performance, especially when used on small files. Accessing memory mapped files is faster than using direct read and write operations for two reasons. Firstly, a system call is orders of magnitude slower than a simple change of program’s local memory. Secondly, in most operating systems the memory region mapped actually is the kernel’s file cache, meaning that no copies need to be created in user space. Using system calls would inevitably involve the time consuming operation of memory copying.

Certain application level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in flat file data storage or configuration files) requires disk access only when a new page boundary is crossed, and can write larger sections of the file to disk in a single operation.

A possible benefit of memory-mapped files is a “lazy loading”, thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but the system only needs to load the smaller page-sized sections as data is being edited, similarly to demand paging scheme used for programs.

http://en.wikipedia.org/wiki/Memory-mapped_file#Benefits

The windows utility to do this is called CreateFileMapping
http://msdn2.microsoft.com/en-us/library/aa366537.aspx

As for multi-processing library the following was recommended:
http://scv.bu.edu/documentation/tutorials/MPI/MPI_text.html

I haven’t found much but the following wikipedia link seems relevant:
http://en.wikipedia.org/wiki/Cluster_(computing)

Defining a Microsoft access Datasource

The more I think about implementation of a GCM the more I think about trying to make the components as independent as possible. I like databases for storing information because they give an orderly framework that is easy to access but they can be slow.

A database is simply a binary file. If we know how information is stored in the file we can optimize ways of retrieving it that could be much faster then a standard database. However, it is nice to have tools that are easy to use for sorting though and organizing information like Microsoft access.

Remember that tables in microsoft access do not need to be stored in the microsoft access database. For instance, you can create linked tables. These linked tables can come for instance from other databases or spreadsheets. All that is really necessary is for Microsoft access to know how to read this information as a table.

This is where I believe data access objects come in. I’ll have to look at them further but I’m hoping that they can be used as a way to use an arbitrary source of information as a table. The following linking gives information on how to create Data Access objects in Microsoft Access.

http://msdn.microsoft.com/en-us/library/ms243192.aspx

I’ll write more once I read about this.

I Decided to Use Com Connect

I searched for Java Com bridges. First I tried google and came across a few programs that looked promising:

http://www.ezjcom.com

http://j-integra.intrinsyc.com/

The interface for ezcom looked nice and there was a free version but I decided to search sourceforge for something free. The two most relevant programs I found were:

http://sourceforge.net/projects/jcom2/

http://jcom2.sourceforge.net/

http://sourceforge.net/projects/comconnect/

http://comconnect.sourceforge.net/tutorial.html

comconnect seemed more relevet for many reaons which include:
-It is still an active project
-It supports both Com and .Net
-The Com application and the Java application don’t need to be on the same computer.

In my search I also found some other interesting bridges:

http://sourceforge.net/projects/propane/

http://propane.sourceforge.net/

http://sourceforge.net/projects/jvm-bridge/

http://semantic.org/jvm-bridge/

Propane is a bridge between java and javascript. I find this interesting because I use firefox a lot which is programed a large part in java script.

The other is a bridge between Haskel and Java.  I find this interesting puerly because I find functional programming languages interesting.  I can not think of an immediate use for it though.

Java with Microsoft Access

I am searching for ways to link Java to Microsoft Access:

——————-Relevant Posts——————–

“There exist several Bridges between Java and COM and I assume that you
can run the Access macros through OLE automation. But where does the
output go? If it is just displayed in the GUI, than you’ll need a way to
presentable it on the web. One way would be to create PDF files

. There
are several programs that allow you to print to a virtual driver that
generates PDF files. Of course you’ll need one that works without user
intervention, most prompt for a file name.”

http://objectmix.com/jdbc-java/41915-can-java-executes-microsoft-access-macro.html

“I believe you can access MS access via the JDBC-ODBC link. Its been along time since tried anything like this. However, most MS work best with other MS products. MS is a company which has to make a profit after all. “

http://forums.sun.com/thread.jspa?threadID=5347786

———-Possibly Relevent Tools———————

“

Quick, easy bridge building. High-level professional quality tool.

Using EZ JCom, developers can call COM/ActiveX components written in any language from Java. Developers can also call Java from COM-aware languages such as C#, Visual Basic or Visual C++.

No knowledge of C++ or COM is necessary.

EZ JCom can be used on Windows platforms to access COM/ActiveX components locally. It can also be used from non-Windows platforms (such as Unix, Linux, Mac, or Handhelds) by using the included Remote Access Service.”

http://javacombridge.com/

Database and Platform Independence

Originally my intent was to use Microsoft access and visual basic as the primary development environment. This is convenient because because Microsoft Access databases are easy to edit and work with. Also visual basic is built into Microsoft access.

When I got thinking about the project further this might be an okay, solution for a personal computer but would not be a good idea for a distributed application. This is because I don’t want to limit the project so that the program can only run on one computer. Therefore, cross platform support is important.

Well, databases have many functionalities, the primary purpose of a database is a store of information. Information can be stored in a database, a spreadsheet, a text file, or a serialized object.  If the information could be potentially stored across many databases, why limit the storage of data just to databases?

The nice thing about Access databases is the built in support for visual basic functions. This makes a database more like an application then just a simple store of information. Databases retrieve information, functions return values, in essence they are not that different. In a way a function or an application, is a generalization of a database. Therefore, rather then think about databases, we should think about entities that provide services.

These entities may have at least one of the following functions

-Retrieve Information

-Refine Information

-Store Information

The entities themselves, can be platform specific, but the arceteture which supports them should not. Therefore, I am thinking of using Java as the primary tool to links these entities together, in a distributed multiplatform fassion.

There will be an abstract class called entityContainer. This class will have functions, that we need to retrieve, store and refine information. One of the information refinement methods will be intialize. The abstract class does not know how information will be intialized but knows the database may need to be intialized.

Other classes will involve the retrieval of  information. For instance, there will be a function called getPressure(coordinate), which will either return the pressure. An isntance of this function may be implemented by calling a macro from inside a database. Alternatively, it could connect to access though an ODB connection. The function may even be implemented to call a perl script file which reads the infromation from a text file.

The Java program could maintain a list of these entity containers in a three dimensional array. Knowing the coordinate, the Java program would select the entity which is most appropriate to retrieve the information. Each entity container could have multiple sources it could retrieve this information from.  The entity container would rank these sources and if it failed to get information from the first source it would try the second source and so on.

Thus at a high level, the Java program would decide what task to preform, well at a lower level separate entities (which could be platform specific and do not need to reside on the same computer) decide how to handle this task.

In this way alot of the data could be built using microsoft access but not limit the overal arcetecuture to a single platform.

Inisght Into Excel

My early idea was to use excel inside of Microsoft Acess in order to display information from the database. I recently was thinking about making, the excel viewer separate from the database. This has the advantage of being able to control multiple climate databases from a single Excel Document.

In previous posts, I described the setup of the database.  The database describes the current state of the earth. There will be two main operations done to the database to try and determine equilibrium.

-The first operation is to initialize the database using current climate information.

-The second is to try and establish equilibrium though iteratively trying to determine the radiation balance point, by point.

There are two ways this latter process might be done:

-The first is to try and figure out radiative balance only for the points being viewed in excel. By this method, the database should iteratively coverage by randomly changing the place where the user decides to view data.

-The second alternative, would try to do the iterative updates in some kind of background thread and update excel at fixed time intervals.

I think the latter option is preferable, however, with regards to the development path, the first option should be easier to implement. Therefore, I think the first option will be implemented first, followed by the seconded option.

I think ultimately, the best option will be to set up some kind of thread management system, where we can simultaneously set up multiple background processes for multiple climate databases. In possible future versions parts of the earth could be portioned into separate databases which could reside on the same or on a separate computer.

The Coordinate System

Given that the surface of the eath is curved one might think that it would be difficult to construct a coordinate system that behaves like a rectangular coordinate system on a small scale but curves around the earth on a large scale.

To approach this problem we ask. How is a sphere like a cube? To a topologist they are nearly identical as it doesn’t take much effort to deform one into the other.

Imagine we had a cubic balloon. We blow it up a little and it is a cube. We draw a separate grid on each face of the cube shaped balloon. Then we blow it up more until it becomes a sphere. If our grid was fine enough then even with the balloon inflated to a spherical shape it still might look like a grid of squares is covering the balloon.

This should give confidence that we can divide the surface of a sphere up into sections that look like squares even though they cover a round object. So to do this we start thinking about what a square balloon might look like if it was inflated into a sphere.  The center of the top face will be mapped to the north pole. The center of the bottom face will be mapped to the south pole, the four faces on the side of the cube will divide the earth into four more sections. And the center point of each of these sections will be mapped to somewhere on the equator.

There are 360 degrees of longitude in the earth. Thus each section covers 90 degrees of longitude. If  we take the prime meridian as one edge then each face adjacent to the prime merridian will be centered at +/-45 degrees longitude.  Additionally the center points of the remain faces will be at +/-135 degrees of longitude.

These results are summarized in the following table. The remaining columns will be discussed in another post:

gridPoints
id	parent	level	left	right	top	bottom	up	down	lat	long	time	P_Theory
1	0	1	5	4	6	3	0	0	90	0	12	0.5
2	0	1	5	4	3	6	0	0	-90	0	12	0.5
3	0	1	5	4	1	2	0	0	0	45	12	0.5
4	0	1	3	6	1	2	0	0	0	135	12	0.5
5	0	1	6	3	1	2	0	0	0	-45	12	0.5
6	0	1	4	5	2	1	0	0	0	-135	12	0.5

The Concept

Should a climate model attempt to predict the weather or is weather far too complex a process to include in a climate model. If we neglect the weather will we miss important cycles such as El Niño and La Niña? There is a temptation in modeling to try to include as many factors as possible to hopefully simulate reality.

However, increasing the complexity of the model increases the potential for error and ads degrees of freedom which are often tuned to fit data rather then taken from fundamental measurement of physics. It is my hope that this blog and the corresponding project

https://sourceforge.net/projects/earthcubed/ (not approved yet):

will succeed by attempting the avoid the complexities inherent in small time step transient models. Instead of focusing on short term transients (i.e. weather) the focus will be more on the global energy balance. This is not to say the model will be simple. The earth will be represented with three special dimensions and two temporal dimensions.

The first temporal dimensions will be the time of day, the seconded temporal dimension will be the day. The first temporal dimension is used to express the steady state solutions.  The second temporal dimension will only be used if dynamic simulations are to be preformed.

The special coordinate system will be built based upon a cube. That is the earths surface will initially be divided up into six curved faces and each division will intersect at right angles.

Further refinement will be done by subdividing each face into grids both vertically and horizontally. The vertical grid will be scaled based on the air density and the horizontal grid will be chosen to make the three dimensional partitions roughly cubical.

With this method the space of the atmosphere will be recursively divided up into a uniformly spaced coordinate grid which will locally behave like a Cartesian coordinate system but globally it will be curved around a cube.