In the age of the world wide web, it is of essential importance that desktop applications can access resources over the internet: they should be able to download files from a web server, write files to an ftp server or read mails from a web server. Often, the ability to access files regardless of their location is called network transparency.

In the past, different approaches to this goals were implemented. The old NFS file system is an attempt to implement network transparency on the level of the POSIX API. While this approach works quite well in local, closely coupled networks, it does not scale for resources to which access is unreliable and possibly slow. Here, asynchronicity is important. While you are waiting for your web browser to download a page, the user interface should not block. Also, the page rendering should not begin when the page is completely available, but should updated regularly as data comes in.

In the KDE libraries, network transparency is implemented in the KIO API. The central concept of this architecture is an IO job. A job may copy, or delete files or similar things. Once a job is started, it works in the background and does not block the application. Any communication from the job back to the application - like delivering data or progress information - is done integrated with the Qt event loop.

Background operation is achieved by starting ioslaves to perform certain tasks. ioslaves are started as separate processes and are communicated with through UNIX domain sockets. In this way, no multi-threading is necessary and unstable slaves can not crash the application that uses them.

File locations are expressed by the widely used URLs. But in KDE, URLs do not only expand the range of addressable files beyond the local file system. It also goes in the opposite direction - e.g. you can browse into tar archives. This is achieved by nesting URLs. For example, a file in a tar archive on a http server could have the URL

http://www-com.physik.hu-berlin.de/~bernd/article.tgz#tar:/paper.tex

In most cases, jobs are created by calling functions in the KIO namespace. These functions take one or two URLs as arguments, and possible other necessary parameters. When the job is finished, it emits the signal result(KIO::Job*). After this signal has been emitted, the job deletes itself. Thus, a typical use case will look like this:

void FooClass::makeDirectory()
{
    SimpleJob *job = KIO::mkdir(KURL("file:/home/bernd/kiodir"));
    connect( job, SIGNAL(result(KIO::Job*)), 
             this, SLOT(mkdirResult(KIO::Job*)) );
}

void FooClass::mkdirResult(KIO::Job *job)
{
    if (job->error())
        job->showErrorDialog();
    else
        cout << "mkdir went fine" << endl;
}

Depending on the type of the job, you may connect also to other signals.

Here is an overview over the possible functions:

Both the KIO::stat() and KIO::listDir() jobs return their results as a type UDSEntry, UDSEntryList resp. The latter is defined as QValueList<UDSEntry>. The acronym UDS stands for "Universal directory service". The principle behind it is that the a directory entry only carries the information which an ioslave can provide, not more. For example, the http slave does not provide any information about access permissions or file owners. Instead, a UDSEntry is a list of UDSAtoms. Each atom provides a specific piece of information. It consists of a type stored in m_uds and either an integer value in m_long or a string value in m_str, depending on the type.

The following types are currently defined:

Although the way of storing information about files in a UDSEntry is flexible and practical from the ioslave point of view, it is a mess to use for the application programmer. For example, in order to find out the MIME type of the file, you have to iterate over all atoms and test whether m_uds is UDS_MIME_TYPE. Fortunately, there is an API which is a lot easier to use: the class KFileItem.

Often, the asynchronous API of KIO is too complex to use and therefore implementing full asynchronicity is not a priority. For example, in a program that can only handle one document file at a time, there is little that can be done while the program is downloading a file anyway. For these simple cases, there is a mucher simpler API in the form of a set of static functions in KIO::NetAccess. For example, in order to copy a file, use

KURL source, target;
source = ...;
target = ...
KIO::NetAccess::copy(source, target);

The function will return after the complete copying process has finished. Still, this method provides a progress dialog, and it makes sure that the application processes repaint events.

A particularly interesting combination of functions is download() in combination with removeTempFile(). The former downloads a file from given URL and stores it in a temporary file with a unique name. The name is stored in the second argument. If the URL is local, the file is not downloaded, and instead the second argument is set to the local file name. The function removeTempFile() deletes the file given by its argument if the file is the result of a former download. If that is not the case, it does nothing. Thus, a very easy to use way of loading files regardless of their location is the following code snippet:

KURL url;
url = ...;
QString tempFile;
if (KIO::NetAccess::download(url, tempFile) {
    // load the file with the name tempFile
    KIO::NetAccess::removeTempFile(tempFile);
}

As can be seen above, the interface to IO jobs is quite abstract and does not consider any exchange of information between application and IO slave that is protocol specific. This is not always appropriate. For example, you may give certain parameters to the HTTP slave to control its caching behavior or send a bunch of cookies with the request. For this need, the concept of meta data has been introduced. When a job is created, you can configure it by adding meta data to it. Each item of meta data consists of a key/value pair. For example, in order to prevent the HTTP slave from loading a web page from its cache, you can use:

void FooClass::reloadPage()
{
    KURL url("http://www.kdevelop.org/index.html");
    KIO::TransferJob *job = KIO::get(url, true, false);
    job->addMetaData("cache", "reload");
    ...
}

The same technique is used in the other direction, i.e. for communication from the slave to the application. The method Job::queryMetaData() asks for the value of the certain key delivered by the slave. For the HTTP slave, one such example is the key "modified", which contains a (stringified representation of) the date when the web page was last modified. An example how you can use this is the following:

void FooClass::printModifiedDate()
{
    KURL url("http://developer.kde.org/documentation/kde2arch/index.html");
    KIO::TransferJob *job = KIO::get(url, true, false);
    connect( job, SIGNAL(result(KIO::Job*)),
             this, SLOT(transferResult(KIO::Job*)) );
}

void FooClass::transferResult(KIO::Job *job)
{
    QString mimetype;
    if (job->error())
        job->showErrorDialog();
    else {
        KIO::TransferJob *transferJob = (KIO::TransferJob*) job;
        QString modified = transferJob->queryMetaData("modified");
        cout << "Last modified: " << modified << endl;
}

When using the KIO API, you usually do not have to cope with the details of starting IO slaves and communicating with them. The normal use case is to start a job and with some parameters and handle the signals the jobs emits.

Behind the curtains, the scenario is a lot more complicated. When you create a job, it is put in a queue. When the application goes back to the event loop, KIO allocates slave processes for the jobs in the queue. For the first jobs started, this is trivial: an IO slave for the appropriate protocol is started. However, after the job (like a download from an http server) has finished, it is not immediately killed. Instead, it is put in a pool of idle slaves and killed after a certain time of inactivity (current 3 minutes). If a new request for the same protocol and host arrives, the slave is reused. The obvious advantage is that for a series of jobs for the same host, the cost for creating new processes and possibly going through an authentication handshake is saved.

Of course, reusing is only possible when the existing slave has already finished its previous job. when a new request arrives while an existing slave process is still running, a new process must be started and used. In the API usage in the examples above, there are no limitation for creating new slave processes: if you start a consecutive series of downloads for 20 different files, then KIO will start 20 slave processes. This scheme of assigning slaves to jobs is called direct. It not always the most appropriate scheme, as it may need much memory and put a high load on both the client and server machines.

So there is a different way. You can schedule jobs. If you do this, only a limited number (currently 3) of slave processes for a protocol will be created. If you create more jobs than that, they are put in a queue and are processed when a slave process becomes idle. This is done as follows:

KURL url("http://developer.kde.org/documentation/kde2arch/index.html");
KIO::TransferJob *job = KIO::get(url, true, false);
KIO::Scheduler::scheduleJob(job);

A third possibility is connection oriented. For example, for the IMAP slave, it does not make any sense to start multiple processes for the same server. Only one IMAP connection at a time should be enforced. In this case, the application must explicitly deal with the notion of a slave. It has to deallocate a slave for a certain connection and then assign all jobs which should go through the same connection to the same slave. This can again be easily achieved by using the KIO::Scheduler:

KURL baseUrl("imap://bernd@albert.physik.hu-berlin.de");
KIO::Slave *slave = KIO::Scheduler::getConnectedSlave(baseUrl);

KIO::TransferJob *job1 = KIO::get(KURL(baseUrl, "/INBOX;UID=79374"));
KIO::Scheduler::assignJobToSlave(slave, job1);

KIO::TransferJob *job2 = KIO::get(KURL(baseUrl, "/INBOX;UID=86793"));
KIO::Scheduler::assignJobToSlave(slave, job2);

...

KIO::Scheduler::disconnectSlave(slave);

You may only disconnect the slave after all jobs assigned to it are guaranteed to be finished.

In the following we discuss how you can add a new ioslave to the system. In analogy to services, new ioslaves are advertised to the system by installing a little configuration file. The following Makefile.am snippet installs the ftp protocol:

protocoldir = $(kde_servicesdir)
protocol_DATA = ftp.protocol
EXTRA_DIST = $(mime_DATA)

The contents of the file ftp.protocol is as follows:

[Protocol]
exec=kio_ftp
protocol=ftp
input=none
output=filesystem
listing=Name,Type,Size,Date,Access,Owner,Group,Link,
reading=true
writing=true
makedir=true
deleting=true
Icon=ftp

The "protocol" entry defines for which protocol this slave is responsible. "exec" is (in contrast what you would expect naively) the name of the library that implements the slave. When the slave is supposed to start, the "kdeinit" executable is started which in turn loads this library into its address space. So in practice, you can think of the running slave as a separate process although it is implemented as library. The advantage of this mechanism is that it saves a lot of memory and reduces the time needed by the runtime linker.

The "input" and "output" lines are not used currently.

The remaining lines in the .protocol file define which abilities the slave has. In general, the features a slave must implement are much simpler than the features the KIO API provides for the application. The reason for this is that complex jobs are scheduled to a couple of subjobs. For example, in order to list a directory recursively, one job will be started for the toplevel directory. Then for each subdirectory reported back, new subjobs are started. A scheduler in KIO makes sure that not too many jobs are active at the same time. Similarly, in order to copy a file within a protocol that does not support copying directly (like the ftp: protocol), KIO can read the source file and then write the data to the destination file. For this to work, the .protocol must advertise the actions its slave supports.

Since slaves are loaded as shared libraries, but constitute standalone programs, their code framework looks a bit different from normal shared library plugins. The function which is called to start the slave is called kdemain(). This function does some initializations and then goes into an event loop and waits for requests by the application using it. This looks as follows:

extern "C" { int kdemain(int argc, char **argv); }

int kdemain(int argc, char **argv)
{
    KLocale::setMainCatalogue("kdelibs");
    KInstance instance("kio_ftp");
    (void) KGlobal::locale();

    if (argc != 4) {
        fprintf(stderr, "Usage: kio_ftp protocol "
                        "domain-socket1 domain-socket2\n");
        exit(-1);
    }

    FtpSlave slave(argv[2], argv[3]);
    slave.dispatchLoop();
    return 0;
}

Slaves are implemented as subclasses of KIO::SlaveBase (FtpSlave in the above example). Thus, the actions listed in the .protocol correspond to certain virtual functions in KIO::SlaveBase the slave implementation must reimplement. Here is a list of possible actions and the corresponding virtual functions:

Additionally, there are reimplementable functions not listed in the .protocol file. For these operations, KIO automatically determines whether they are supported or not (i.e. the default implementation returns an error).

All these implementation should end with one of two calls: If the operation was successful, they should call finished(). If an error has occurred, error() should be called with an error code as first argument and a string in the second. Possible error codes are listed as enum KIO::Error. The second argument is usually the URL in question. It is used e.g. in KIO::Job::showErrorDialog() in order to parameterize the human-readable error message.

For slaves that correspond to network protocols, it might be interesting to reimplement the method SlaveBase::setHost(). This is called to tell the slave process about the host and port, and the user name and password to log in. In general, meta data set by the application can be queried by SlaveBase::metaData(). You can check for the existence of meta data of a certain key with SlaveBase::hasMetaData().

Various actions implemented in a slave need some way to communicate data back to the application using the slave process:

Sometimes a slave has to interact with the user. Examples include informational messages, authentication dialogs and confirmation dialogs when a file is about to be overwritten.