From: jeffj@panix.com (Jeff Jonas) Newsgroups: alt.folklore.computers Subject: mmapp for file copy Date: 10 Mar 2000 20:00:32 -0500 Organization: Jeff's House of Electronic Parts Lines: 55 Message-ID: <8ac5rg$hgu$1@panix.com> References: <38C74978.43F36FF8@netinsight.se> <8abehh$gft$1@agate.berkeley.edu> <952723349.1322@shelley.paradise.net.nz> NNTP-Posting-Host: panix.com X-Trace: news.panix.com 952736432 17727 166.84.0.226 (11 Mar 2000 01:00:32 GMT) X-Complaints-To: abuse@panix.com NNTP-Posting-Date: 11 Mar 2000 01:00:32 GMT X-Newsposter: trn 4.0-test55 (26 Feb 97) Path: chonsp.franklin.ch!pfaff.ethz.ch!news-zh.switch.ch!news-ge.switch.ch!news.maxwell.syr.edu!netnews.com!news!news-peer.gip.net!news.gsl.net!gip.net!panix!news.panix.com!not-for-mail Xref: chonsp.franklin.ch alt.folklore.computers:51575 >>>You want to DMA the data from the disk to memory, and the DMA the >>>data from memory to disk again. *WELL* if you were *really* efficient about it, the data would be done entirely by the disk controller and never get so far as the main RAM/core! With SCSI you'd do a block transfer direct from device to device, the data flowing thru the SCSI bus amid the devices, or bouncing back to the same disk but to different blocks! Masscomp (later merged into Concurrent Computer) had real time Unix. One of the facilities pre-allocated disk space so the files LOOKED like a Unix file, but the disk blocks were passed to I/O controllers so data acquisition hardware would DMA their data direct to disk. When done, read the file! One would suppose that making a list of source and destination data blocks would allow a VERY direct copying with the data never going off the controller card! I was under the impression that mainframes had such knowledge embedded in many of their utilties, thus the confusing maze of choosing the best file copy/generation utility for your particular need. There have been many intersting followups concerning the mmap solution. While it seems counter-intuitive at first, it's quite ingenious. The traditional while (read (... buffer ...) write (... buffer ...) has drawbacks: 1) overhead of 2 system calls per buffer moved 2) overhaed of copying the buffer from system buffer to user's space just to copy back to a buffer, unaltered. 3) how long should the buffer be? Unless you use exact page sizes, it's only guessing 4) if the buffer's not page aligned, the copyin/copyout between the user process and kernel could take longer. In the least it's not neat! By using mmap, the virtual memory system's doing all the work. If you let mmap() choose the address to where it attaches, it'll naturally be page aligned and naturally read/write entire pages behind the scenes. Ideally the data would be read/written direct from/to the cache without intermediate copying. That's a big win: letting the system handle the addresses for you. Admittedly, mmap has upper limits: you must specify the length and offset when mapping the input file into memory space. BUT if the file size > address space, then just repeat with a non-zero offset and copy a HUGE chunk at a time! -- Jeffrey Jonas jeffj@panix(dot)com The original Dr. JCL and Mr .hide ###### Sender: Ian Stirling From: Ian Stirling Subject: Re: mmapp for file copy Newsgroups: alt.folklore.computers References: <38C74978.43F36FF8@netinsight.se> <8abehh$gft$1@agate.berkeley.edu> <952723349.1322@shelley.paradise.net.nz> <8ac5rg$hgu$1@panix.com> Organization: None.. User-Agent: tin/pre-1.4-980818 ("Laura") (UNIX) (Linux/2.2.13 (i586)) Lines: 18 Message-ID: X-Abuse-Info: Please be sure to forward a copy of ALL headers X-Abuse-Info: Otherwise we will be unable to process your complaint properly X-Complaints-To: admin@usenetserver.com NNTP-Posting-Date: Sat, 11 Mar 2000 11:57:52 EST Date: Sat, 11 Mar 2000 16:57:52 GMT Path: chonsp.franklin.ch!pfaff.ethz.ch!news-zh.switch.ch!news-ge.switch.ch!news.maxwell.syr.edu!cyclone1.usenetserver.com!cyclone1.usenetserver.com!news-west.usenetserver.com.POSTED!not-for-mail Xref: chonsp.franklin.ch alt.folklore.computers:51664 Jeff Jonas wrote: >>>>You want to DMA the data from the disk to memory, and the DMA the >>>>data from memory to disk again. >*WELL* if you were *really* efficient about it, the data would be >done entirely by the disk controller and never get so far as the >main RAM/core! With SCSI you'd do a block transfer direct from >device to device, the data flowing thru the SCSI bus amid the >devices, or bouncing back to the same disk but to different blocks! Now, for real fun, work out which devices have firmware bugs in this poorly excersised code path. -- http://inquisitor.i.am/ | mailto:inquisitor@i.am | Ian Stirling. ---------------------------+-------------------------+-------------------------- Things a surgeon should never say: Better save that for the autopsy.