Sharing allocated memory across multiple jobs
Last Post 05 Dec 2012 03:10 PM by Rory Hewitt. 11 Replies.
AddThis - Bookmarking and Sharing Button Printer Friendly
  •  
  •  
  •  
  •  
  •  
Sort:
PrevPrev NextNext
You are not authorized to post a reply.
Author Messages
Rory Hewitt
Veteran Member
Veteran Member
Posts:1311
Avatar

--
04 Dec 2012 12:26 PM

All,

I am trying to figure out the best way to 'allocate memory' in one job that can be used by another job. I could obviously have the 'allocating' job create a user space in a library (not QTEMP) and then somehow let the other job know the library/name in which the user space was created. However, my processing could potentially result in thousands of (possibly small) allocations, so I'm concerned about overhead etc.

Does anyone have any experience of using shared memory (from RPG)? Might that work? Failing that, are there any API's I can use to allocate memory (possibly teraspace-size allocations) that is both available to other jobs and which is still available after the job which allocated it ends? I'm thinking of the MI space operations (http://publib.boulder.ibm.com/iseri...tm#minsm).

Thanks,

Rory

benderd
Advanced Member
Advanced Member
Posts:596

--
04 Dec 2012 03:13 PM
Rory,
not knowing what you are trying to do...
- the classic approach: make a file storage resident with SETOBJACC
- the single level store approach: just use any persistent object to share memory, if it's used by many, it will stay resident
- the client server approach: having a NEP providing setters and getters for information sitting on a dtaq or something similar _ I would implement this in a language optimized for this stuff, like Java (=> AppServer4RPG)

D*B
Access any Database from ADABAS to XBASE with embedded SQL in RPG. http://sourceforge.net/projects/appserver4rpg/
Rory Hewitt
Veteran Member
Veteran Member
Posts:1311
Avatar

--
04 Dec 2012 04:09 PM
Dieter,

It's basically for a generic session manager process for CGI - when a CGI program is first called, it requests memory from a controller. It then uses that memory to hold session- and transaction-specific data. However, since the CGI program might be called in different jobs, depending on the HTTP server load, it needs to be able to get access to 'its' memory whenever it's called. My controller stores session 'control' information in a user space, but the actual program data requested by the CGI program obviously won't go in there.

I'd prefer to keep the data separate from any permanent objects, as it could contain sensitive data, hence my oblique comments about 'allocating' memory. as a last resort, Id use user spaces.

I have my own NEP processing on a user queue (not a data queue), but I don't think it's appropriate for this task, for a variety of reasons. I've looked at your AppServer4RPG, but I found the performance of my app worked fine for my needs.

benderd
Advanced Member
Advanced Member
Posts:596

--
05 Dec 2012 12:33 AM
Rory,

why don't you use a running process per session, sitting on a Q, communicating with the CGI process? it might be a little bit more expensive at startup tine, but this could be optimized by some sort of pooling...

D*B (trying to reinvent wheels)
Access any Database from ADABAS to XBASE with embedded SQL in RPG. http://sourceforge.net/projects/appserver4rpg/
davesmith
Veteran Member
Veteran Member
Posts:1243
Avatar

--
05 Dec 2012 03:29 AM
Rory,

I'm not sure I understand enough about what you're doing to comment whether this is the best approach or not, but there is an implementation of the unix shared memory functionality, that should be able to do pretty much what you described in your first post. ie.

http://publib.boulder.ibm.com/infoc...nix3a3.htm

Obviously if performance is important you'll need to write some form of pooling mechanism around this so every CGI session doesnt result in allocation/deallocation of these segments. Which along with writing the access serialisation logic, makes it non-trivial to implement of course, but it should work effectively along the lines you describe.

Dave
john erps
New Member
New Member
Posts:9

--
05 Dec 2012 04:12 AM
Rory,

Maybe you could use persistent CGI, in combination with named activation groups.

It seems you try to implement "persistent CGI" functionality, which is provided out of the box.
Emmanuel
Veteran Member
Veteran Member
Posts:778
Avatar

--
05 Dec 2012 07:05 AM

Maybe I'm way off base, but isn't this what subsystems do? (WRKSHRPOOL).

 Cheers,

Emmanuel

Ringer
Veteran Member
Veteran Member
Posts:1725
Avatar

--
05 Dec 2012 07:22 AM
I can build an extending user space of 64K in 0.007 seconds. 1 Meg in size in about 0.013 seconds, 16 Meg in 0.15 seconds. So I don't think that would a bottleneck. I'd create a user space on-the-fly one time for the client (for that session), encrypt any sensitive data (AES ciper?) in it, hash any one way data like a password (SHA256?) and build the user spaces as *PUBLIC *EXCLUDE in a library flagged as *PUBLIC *EXCLUDE. Have the CGI (RPG/CL) adopt authority so they can access the user space. Pass back the user space name as a HTTP header cookie (~session ID).

I think CGIDEV2 may already have sub-procedures to some of this. Just my thoughts.

Chris Ringer
Henrik Rutzou
Advanced Member
Advanced Member
Posts:596
Avatar

--
05 Dec 2012 09:36 AM

The easy way is to have a session number that are passed on every call and then a DB2 Table that has

Session number (key)

Variable name (key)

Variable value 


and then code a service program with subprocedures the can either put og get a variable or delete all variables from a session

Rory Hewitt
Veteran Member
Veteran Member
Posts:1311
Avatar

--
05 Dec 2012 10:48 AM
All,

Thanks for your responses.

I actually already implemented the shared memory processing, similar to the one that Dave pointed to. It works very well (and fast) and seems to do everything I want. Access serialization is handled using this: http://www.iprodeveloper.com/articl...tin-699257 - easy!

I would rather not use Persistent CGI because of all the other hassles that it entails. PCGI always seemed like the Holy Grail, but then when you delve into it, it's just horribly implemented. I've never heard of anyone doing a truly successful implementation - john erps, have you managed it?

Chris's point about user spaces is well taken (that performance may not really be a problem), but I'm still concerned about the security issues, as well as the potential proliferation of user spaces.

Henrik, your idea is basically what I'm trying to get round - we currently have this sort of thing in place, but I would like the ability to simply allow the CGI program to say "Give me 10,000 bytes and return me a pointer to it) and then, within that, do whatever it wants (which might well not be as simple as storing individual variables).

This is needs to be generic, as it is intended to be a utility available for use by almost anyone. As such, I want to make it both 'neat' and 'simple'. Thus, the easiest approach (from the POV of building the software) might not be best for the eventual end-users.

Rory
Henrik Rutzou
Advanced Member
Advanced Member
Posts:596
Avatar

--
05 Dec 2012 02:46 PM

Rory,

this may not be so simple if you want the jobs to update the storage and can't lock the memory while the QZSRCGI runs and you are calling to the server in nested AJAX calls.

Lets say you fire up 5 AJAX calls to the server in a javascript, the javascript will not execute the javascript sequential but will fire up 5 new server HTTP connections and thereby sessions without the javascript waits for one call to finish before it starts the nest. This will cause the calls to use 5 QZSRCGI jobs that runs parallel at the same time.

If these QZSRCGI jobs shares common storage and update the storage you get unpredictable results. 

The same goes for PCGI because a session is not what we understand with a session - Persistancy is based on the HTTP server and the CGIjob holds the HTTP connnection open. If you open a new browser window you have then two open HTTP connections and thereby two CGIjob bound (like in 5250) to these connections - however AJAX work in the exactly same way as several browser sessions they open new HTTP connections because the first A in AJAX stands for Asyncrone


.    

 


Rory Hewitt
Veteran Member
Veteran Member
Posts:1311
Avatar

--
05 Dec 2012 03:10 PM
Henrik,

A very good point, and a nice example. The assumption (and I know one shouldn't make assumptions) is that simple Ajax calls wouldn't involve the data being updated. Of course, even if it does, my serialization would cater for that (because the locksl processing would lock the memory to each job until that job has finished with it), but of course, that would negate one of the benefits of Ajax, since then the threads would effectively run concurrently.

The shared memory (or whatever) which would be 'controlled' by this method would typically be updated on a non-Ajax POST. so a job could use multiple Ajax calls to retrieve data, but it would be a final POST which would 'write' that data to the program.

Rory
You are not authorized to post a reply.

Acceptable Use Policy