			Implementation Details
			======================

			Author: Stewart Brodie
			Date: 29th October '97




0.  Document history
1.  Introduction
2.  "Session.headers"
2.1    During Requests
2.2    During Responses
3.  Reading data
4.  Chunked data transfers
5.  Data buffers
6.  Counters
7.  Errors
8.  Compression
9.  Build options




0.  Document History

Date     Issue  Comments

29/10/97  1     First version
21/04/98  2     Notes about build options added and compression



1.  Introduction


The HTTP module is one of the more complex fetcher modules.  It performs many
different operations required by the HTTP specifications and attempts to
fulfil them reliably and in a manner which is non-destructive to remote
servers.  This document details *some* of the implementation decisions which
somebody maintaining this module may need to be aware of.

HTTP fetches are started with a call to SWI HTTP_GetData (via URL_GetURL).
Then the client application repeatedly calls ReadData and Status (again
via the URL module) until all the data is returned, and then HTTP_Stop (via
URL_DeregisterURL or URL_Stop) to clean up.

This is the standard fetcher interface, however, many things are done 
internally to optimise use of resources, which are described below.

If the AcornSSL module is loaded, then HTTPS fetches are started with
a call to SWI HTTP_SecureGetData (again via URL_GetURL).  The SWI names
match those of normal HTTP except with the word "Secure" as a prefix of
the SWI name.  These SWIs are based at SWI_base+&10.



2.  "Session.headers"


The Session data structure contains a "headers" member which is used for
accumulating HTTP headers.  This is used for BOTH the request headers and
the response headers.


2.1  During Requests

When the request is being formulated (during HTTP_GetData), headers are added
to the list with "http_add_header".  This function adds headers to a header
list (the function itself is a generic list builder).  The header list can be
thought of as an associative array  (duplicate indices allowed) which is kept
in the order in which items were added.  Routines are provided for locating a
header, deleting a header, deleting all the headers.  Once all the standard
headers have been added, attention turns to the client-supplied data (pointer
in R4 on entry). These are also parsed and any entity-body in the client data
is then duplicated into Session.data (length in Session.data_len).

Once the list is complete, the headers are all examined and inappropriate
ones are removed.  If the method is an idempotent read (GET or HEAD), any
entity body is discarded completely and related headers are also ditched
(Content-Length, Content-Type, Transfer-Encoding).  This can help clients
which "forget" to zero R4 (eg. !Browse (as of October 1997) on requests
generated by a redirect respose to a POST).

If the method was a POST or PUT, then a content-length header is sought, and
is added if it was missing.  If it was already there but indicated an
amount of data exceeding Session.data_len, it is rewritten to match
Session.data_len.

Finally, a buffer is created and filled with the complete request buffer
and stored in Session.fullreq.  The act of pushing everything into the buffer
also EMPTIES the Session.headers list.  The first "header" line is separated
from its "value" by a space, subsequent lines use a colon.  The first
"header" is actually the HTTP method being used (eg. "GET" or "POST") and the
value is the URI plus a space plus the HTTP version token - hence the reason
for the special treatment of the first line.


2.2  During Responses

A similar process is followed when reading the response.  Whilst the headers
are still being retrieved (Session.donehead is zero or Session.chunking is
non-zero and Session.chunkstate is reading headers or footers) a COPY of the
data ready on an incoming socket is requested (recv is called with MSG_PEEK
in the fourth parameter) and written into an internally allocated buffer. 
Headers are parsed from this just like they were from the client data on a 
request - it uses the same routine.  Content-length is trapped and parsed,
the value stored and the header then DISCARDED.

Once all the headers are known, then they are filtered as appropriate. 
Content-length is ignored if there was a transfer-encoding.  The HTTP version
is reset to 1.0 for the benefit of the client.  Transfer-Encoding is parsed
to look for the string "chunked" which sets Session.chunking to non-zero and
the header is then removed.  Connection headers are removed and any headers
matching tokens in the connection header are also removed (they do not
concern our client, as they refer solely to characteristics of the connection
between this module and the remote server).  Incoming headers are sanitised
(continuation lines are rejoined, and leading/trailing spaces are removed and
generally tidied up).  Cookies, if compiled in (need -DCOOKIE in Makefile),
are then parsed and stored if appropriate.  However, they are not removed
from the header list.

After all of that, if not chunking, the content-length header is added back
to the response headers but at the end of the header list, and a buffer is
generated containing the complete response header.  Thus not all of the
headers will be seen by the client, and those that are present are not
necessarily in the same order that the server sent them.  This should not
concern the client.

As each header line is read from the buffer (including reconstructing
split lines), the data representing that header is read and discarded from
the TCP/IP stack (recv is called again but without MSG_PEEK).  (Actually, the
header parser keeps a running total of how much data has been "consumed", and
only discards the data in one go when it has processed as many headers as it
could.




3.  Reading data


Headers are read into a private buffer (Session.buffer).  When reading body
data, data is read directly into the client's buffer.  The function 
"http_write_data_to_client" must always called in order to update R2, R3 and
R4 when client data has been generated and that function copies data into the
client buffer only if it needs to.

R2, R3 and R4 are updated so that future calls to write more data are
appended to the buffer (if they occur during the execution of the same SWI
call, of course).  This can be done because the wrapper function
http_readdata enforces the spec requirement that R2 and R3 be preserved
across the call.  Currently, the only time when data will be written to the
client buffer in multiple calls to this function is when the end of a "chunk"
has been reached and there is room to also store the start of the next chunk
too.


4.  Chunked data transfers


The header reading & parsing code is used to parse the chunk-size
declarations although the Session.headers structure is not used to store the
data.  When in "data reading" mode, data is read directly to the client
buffer, although the size of the transfer is limited to the minimum of the
buffer size and the amount of data left in the current chunk.  This
simplifies the processing of chunked transfers greatly, since you never have
to worry about having to not put chunking headers in the client's buffer.

The conditions in the ReadData handler are made much more complex by the
chunking checks, but unless lots of code is to be duplicated, this cannot be
helped.


5.  Data buffers

All of the data buffers allocated by the module are kept in the RMA.  They
are managed with malloc/free/module_realloc.  module_realloc is a wrapper to
the C library's realloc which is broken in the RISC OS 3.1 ROM build.  The
code expects "broken_os" to be set non-zero if it needs to call OS_Module 13
directly instead of relying on the library.  "broken_os" is initialised with
the truth value of "R1 result of OS_Byte 129,0,255" <= &A4



6.  Counters

Session.sent indicates the amount of entity-body sent to the client, and NOT
the total number of bytes provided to the client.  This is critical otherwise
you can accidentally cut off some transfers prematurely as explained here:

(All lengths are arbitrarily chosen numbers to illustrate the problem). A
client requests a URL, goes through all the startup and then calls the
ReadData SWI with details of its buffer - which is 8192 bytes long.  The
module receives a block of data from the remote server which is 8192 bytes
long.  The headers take up the first 1000 bytes, and the next 7192 bytes are
the first part of the object being transferred.  The content-length of the
object being downloaded specifies its size as 8000 bytes.  If we write the
headers and first 7192 bytes of the object to the clients buffer, then if we
include the header size in the Session.sent count, then when the exit value
for R5 is calculated, it will notice that the 8192 bytes which have been
"sent" exceeds (or matches) the object's content-length, hence it will
determine that the transfer is complete and set R5 to zero - which the client
is watching for to indicate completion.  Thus we "lose" 808 bytes off the end
of the object.  By ignoring the header size, R5 will be set correctly to 808
and the browser will continue to poll for the remaining data.



7.  Errors

Are all stored in the Messages file and read into a static _kernel_oserror
buffer at run-time.



8.  Compression

The HTTP protocol has the ability to use on-the-fly compression of entity
bodies being transferred in order to decrease used bandwidth.  If the
compression code is enabled (see Build Options) then on-the-fly decompression
will be performed, but, obviously, only in transfers where the server has
noticed our declaration of decompress capability and taken advantage of it.


9.  Build Options

Several features of the AcornHTTP module can be controlled through the use
of pre-processor macros.  These are set in the Makefile in the DFLAGS macro.
Currently, only COOKIE and COMPRESSION should be altered.

COMPRESSION controls the ability to handle compressed transfers as described
above.  COOKIE controls the ability to process cookies.  If an application
is running in an environment where it wishes to handle all the cookie stuff
itself, it can do so by using a build of this module which does not include
the cookie code.

