Remote Services

Pelix/iPOPO provides support for remote services, i.e. consuming services provided from another framework instance. This provider can run on the same machine as the consumer, or on another one.

Concepts

Pelix/iPOPO remote services implementation is a based on a set of services. This architecture eases the development of new providers and allows to plug in or update protocols providers at run time.

In this section, we will shortly describe the basic concepts of Pelix Remote Services, i.e.:

  • the concept of import and export endpoints
  • the core services required to activate remote services
  • the discovery providers
  • the transport providers

The big picture of the Pelix Remote Services can be seen as:

Architecture of Pelix Remote Services

Note that Pelix Remote Services implementation has been inspired from the OSGi Remote Services specification, and tries to reuse most of its constants, to ease compatibility.

Before that, it is necessary to see the big picture: how does Pelix Remote Services works.

How does it work?

The export and import of a service follows this sequence diagram, described below:

Sequence of the export and import of a service

When a service declares it can be exported, the export dispatcher detects it (as it is a service listener) notifies all transport providers which matches the service properties. Each transport provider then tests if it can/must create an endpoint for it and, if so, returns an export endpoint description to the exports dispatcher. The endpoint implementation is transport-dependent: it can be a servlet (HTTP-based procotols), a serial-port listener, ... As a result, there can be multiple export endpoints for a single service: (at least) one per transport provider. The description of each export endpoint is then stored in the exports dispatcher, one of the core services of Pelix Remote Services.

When an endpoint (or a set of endpoints) is stored in the exports dispatcher, the discovery providers are notified and send there protocol-specific events. They can target other Pelix frameworks, but also any other kind of frameworks (OSGi/Java, ...) or of software (like a Node.js server with mDNS support). Those events indicate that new export endpoints are available: they can point to the description of this endpoint or contain its serialized form. Note that the description sent over the network must be an import-side description: it should contain all information required to connect and use the endpoint, stored in import properties so that the newly imported services don’t get exported by mistake.

Another framework using the same discovery provider can capture this event and handle the new set of import endpoints. Those endpoints will be stored in the imports registry, the other core service of Pelix Remote Services. If multiple discovery providers find the same endpoints, don’t worry, they will be filtered out according to their unique identifier (UUID).

The imports registry then notifies the transport providers to let them create a local proxy to the remote service and register it as a local service (with import properties). This remote service is now usable by local consumers.

Note

In the current implementation of Pelix Remote Services, the same remote service can be imported multiple times by the same consumer framework. This is due to the fact that the imported service is created by the transport providers and not by the centralized imports registry.

This behaviour is useful when you want to consume a service from a specific provider, or if you can sort transport providers by efficiency. This has to been taken into account in some cases, like when consuming multiple services of the same specification while multiple transport providers are active.

This behaviour is subject to debate but is also used in some projects. It could be modified if enough problems are reported either on the mailing list or in GitHub issues.

Finally, Pelix Remote Services also supports the update of service properties, which can be handled as a minimalist event by the discovery providers, e.g. containing only the endpoint UID and the new properties. The unregistration is often the simplest event of a discovery provider, sending only the endpoint UID.

Export/Import Endpoints

The endpoints objects are declared in pelix.remote.beans by the ExportEndpoint and ImportEndpoint classes.

Both contain the following information:

  • UID: the unique identifier of the endpoint. It is a class-4 UUID, which should be unique across frameworks.
  • Framework: the UID of the framework providing the endpoint. It is mainly used to clean up the endpoints of a lost framework. If too many endpoint UID collisions are reported, it could be used as a secondary key.
  • Name: the name of the endpoint. It can have a meaning for the transport provider, but isn’t used by Pelix itself.
  • Properties: a copy of the current properties of the remote service.
  • Specifications: the list of service exported specifications. A service can choose to export a subset of its specifications, as some could be private or using non-serializable types.
  • Configurations: the list of transports allowed to export this endpoint or used for importing it.

Finally, the ExportEndpoint object also gives access to the service reference and implemnetation, in order to let transport providers access the methods and properties of the service.

Core Services

The core services of the Pelix Remote Services implementation is based on two services:

  • the exports dispatcher which keeps track of and notifies the discovery providers about the export endpoints created/updated/deleted by transport providers. If a discovery provider appears after the creation of an export endpoint, it will still be notified by the exports dispatcher.

    This service is provided by an auto-instantiated component from the pelix.remote.dispatcher bundle. It provides a pelix.remote.dispatcher service.

  • the imports registry which keeps track of and notifies the transports providers about the import endpoints, according to the notifications from the discovery providers. If a transport provider appears after the registration of an import endpoint, it will nevertheless be notified by the imports registry of existing endpoints.

    This service is provided by an auto-instantiated component from the pelix.remote.registry bundle. It provides a pelix.remote.registry service.

Dispatcher Servlet

The content of the exports dispatcher can be exposed by the dispatcher servlet, provided by the same bundle as the exports dispatcher, pelix.remote.dispatcher. Most discovery providers rely on this servlet as it allows to get the list of exported endpoints, or the details of a single one, in JSON format.

This servlet must be instantiated explicitly using its pelix-remote-dispatcher-servlet-factory factory. As it is a servlet, it requires the HTTP service to be up and running to provide it to clients.

Its API is very simple:

  • /framework: returns the framework UID as a JSON string
  • /endpoints: returns the whole list of the export endpoints registered in the exports dispatcher, as a JSON array of JSON objects.
  • /endpoint/<uid>: returns the export endpoint with the given UID as a JSON object.

Discovery Providers

A framework must discover a service before being able to use it. Pelix/iPOPO provides a set of discovery protocols:

  • a home-made protocol based on UDP multicast packets, which supports addition, update and removal of services;
  • a home-made protocol based on MQTT, which supports addition, update and removal of services;
  • mDNS, which is a standard but doesn’t support service update;
  • a discovery service based on Redis.

Transport Providers

The remote services implementation supports XML-RPC (using the xmlrpc standard package), but it is recommended to use JSON-RPC instead (using the jsonrpclib-pelix third-party module). Indeed, the JSON-RPC layer has a better handling of dictionaries and custom types. iPOPO also supports a variant of JSON-RPC, Jabsorb-RPC, which adds Java type information to the arguments and results. As long as a Java interface is correctly implementing, this protocol allows a Python service to be used by a remote OSGi Java framework, and vice-versa. The OSGi framework must host the Java implementation of the Pelix Remote Services.

All those protocols require the HTTP service to be up and running to work. Finally, iPOPO also supports a kind of MQTT-RPC protocol, i.e. JSON-RPC over MQTT.

Providers included with Pelix/iPOPO

This section gives more details about the usage of the discovery and transport providers included in Pelix/iPOPO. You’ll need at least a discovery and a compatible transport provider for Pelix Remote Services to work.

Apart MQTT, the discovery and transport providers are independent and can be used with one another.

Multicast Discovery

Bundle:pelix.remote.discovery.multicast
Factory:pelix-remote-discovery-multicast-factory
Requires:HTTP Service, Dispatcher Servlet
Libraries:nothing (based on the Python Standard Library)

Pelix comes with a home-made UDP multicast discovery protocol, implemented in the pelix.remote.discovery.multicast bundle. This is the original discovery protocol of Pelix/iPOPO and the most reliable one in small local area networks. A Java version of this protocol is provided by the Cohorte Remote Services implementation.

This protocol consists in minimalist packets on remote service registration, update and unregistration. They mainly contain the notification event type, the port of the HTTP server of the framework and the path to the dispatcher servlet. The IP of the framework is the source IP of the multicast packet: this allows to get a valid address for frameworks on servers with multiple network interfaces.

This provider relies on the HTTP server and the dispatcher servlet. It doesn’t have external dependencies.

The bundle provides a pelix-remote-discovery-multicast-factory iPOPO factory, which must be instantiated to work. It can be configured with the following properties:

Property Default value Description
multicast.group 239.0.0.1 The multicast group (address) to join to send and receive discovery messages.
multicast.port 42000 The multicast port to listen to

To use this discovery provider, you’ll need to install the following bundles and instantiate the associated components:

# Start the HTTP service with default parameters
install pelix.http.basic
start $?
instantiate pelix.http.service.basic.factory httpd

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Instantiate the dispatcher servlet
instantiate pelix-remote-dispatcher-servlet-factory dispatcher-servlet

# Install and start the multicast discovery with the default parameters
install pelix.remote.discovery.multicast
start $?
instantiate pelix-remote-discovery-multicast-factory discovery-mcast

mDNS Discovery

Bundle:pelix.remote.discovery.mdns
Factory:pelix-remote-discovery-zeroconf-factory
Requires:HTTP Service, Dispatcher Servlet
Libraries:pyzeroconf

The mDNS protocol, also known as Zeroconf, is a standard protocol based on multicast packets. It provides a Service Discovery layer (mDNS-SD) based on the DNS-SD specification.

Unlike the home-made multicast protocol, this one doesn’t support service updates and gives troubles with service unregistrations (frameworks lost, ...). As a result, it should be used only if it is required to interact with other mDNS devices.

In order to work with the mDNS discovery from the Eclipse Communication Framework, the pyzeroconf library must be patched: the .local. check in zeroconf.mdns.DNSQuestion must be removed (around line 220).

This provider is implemented in the pelix.remote.discovery.mdns bundle, which provides a pelix-remote-discovery-zeroconf-factory iPOPO factory, which must be instantiated to work. It can be configured with the following properties:

Property Default value Description
zeroconf.service.type _pelix_rs._tcp.local. Zeroconf service type of exported services
zeroconf.ttl 60 Time To Live of services (in seconds)

To use this discovery provider, you’ll need to install the following bundles and instantiate the associated components:

# Start the HTTP service with default parameters
install pelix.http.basic
start $?
instantiate pelix.http.service.basic.factory httpd

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Instantiate the dispatcher servlet
instantiate pelix-remote-dispatcher-servlet-factory dispatcher-servlet

# Install and start the mDNS discovery with the default parameters
install pelix.remote.discovery.mdns
start $?
instantiate pelix-remote-discovery-zeroconf-factory discovery-mdns

Redis Discovery

Bundle:pelix.remote.discovery.redis
Factory:pelix-remote-discovery-redis-factory
Requires:nothing (all is stored in the Redis database)
Libraries:redis

The Redis discovery is the only one working well in Docker (Swarm) networks. It uses a Redis database to store the host name of each framework and the description of each exported endpoint of each framework. Those description are stored in the OSGi standard EDEF XML format, so it should be possible to implement a Java version of this discovery provider. The Redis discovery uses the key events of the database to be notified by the latter when a framework or an exported service is registered, updated, unregistered or timed out, which makes it both robust and reactive.

This provider is implemented in the pelix.remote.discovery.redis bundle, which provides a pelix-remote-discovery-redis-factory iPOPO factory, which must be instantiated to work. It can be configured with the following properties:

Property Default value Description
redis.host localhost The hostname of the Redis server
redis.port 46379 The port the Redis server listens to
redis.db 0 The Redis database to use (integer)
redis.password None Password to access the Redis database
heartbeat.delay 10 Delay in seconds between framework heart beats

To use this discovery provider, you’ll need to install the following bundles and instantiate the associated components:

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Install and start the Redis discovery with the default parameters
install pelix.remote.discovery.redis
start $?
instantiate pelix-remote-discovery-redis-factory discovery-redis

XML-RPC Transport

Bundle:pelix.remote.xml_rpc
Factories:pelix-xmlrpc-exporter-factory, pelix-xmlrpc-importer-factory
Requires:HTTP Service
Libraries:nothing (based on the Python Standard Library)

The XML-RPC transport is the first one having been implemented in Pelix/iPOPO. Its main advantage is that is doesn’t depend on an external library, XML-RPC being supported by the Python Standard Library.

It has some troubles with complex and custom types (dictionaries, ...), but can be used without problems on primitive types. The JSON-RPC transport can be preferred in most cases.

Like most of the transport providers, this one is split in two components: the exporter and the importer. Both must be instantiated manually.

The exporter instance can be configured with the following property:

Property Default value Description
pelix.http.path /XML-RPC The path to the XML-RPC exporter servlet

To use this transport provider, you’ll need to install the following bundles and instantiate the associated components:

# Start the HTTP service with default parameters
install pelix.http.basic
start $?
instantiate pelix.http.service.basic.factory httpd

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Install and start the XML-RPC importer and exporter with the default
# parameters
install pelix.remote.xml_rpc
start $?
instantiate pelix-xmlrpc-exporter-factory xmlrpc-exporter
instantiate pelix-xmlrpc-importer-factory xmlrpc-importer

JSON-RPC Transport

Bundle:pelix.remote.json_rpc
Factories:pelix-jsonrpc-exporter-factory, pelix-jsonrpc-importer-factory
Requires:HTTP Service
Libraries:jsonrpclib-pelix (installation requirement of iPOPO)

The JSON-RPC transport is the recommended one in Pelix/iPOPO. It depends on an external library, jsonrpclib-pelix which has no transient dependency. It has way less troubles with complex and custom types than the XML-RPC transport, which eases the development of most of Pelix/iPOPO applications.

Like most of the transport providers, this one is split in two components: the exporter and the importer. Both must be instantiated manually.

The exporter instance can be configured with the following property:

Property Default value Description
pelix.http.path /JSON-RPC The path to the JSON-RPC exporter servlet

To use this transport provider, you’ll need to install the following bundles and instantiate the associated components:

# Start the HTTP service with default parameters
install pelix.http.basic
start $?
instantiate pelix.http.service.basic.factory httpd

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Install and start the JSON-RPC importer and exporter with the default
# parameters
install pelix.remote.json_rpc
start $?
instantiate pelix-jsonrpc-exporter-factory jsonrpc-exporter
instantiate pelix-jsonrpc-importer-factory jsonrpc-importer

Jabsorb-RPC Transport

Bundle:pelix.remote.transport.jabsorb_rpc
Factories:pelix-jabsorbrpc-exporter-factory, pelix-jabsorbrpc-importer-factory
Requires:HTTP Service
Libraries:jsonrpclib-pelix (installation requirement of iPOPO)

The JABSORB-RPC transport is based on a variant of the JSON-RPC protocol. It adds Java typing hints to ease unmarshalling on Java clients, like the Cohorte Remote Services implementation. The additional information comes at small cost, but this transport shouldn’t be used when no Java frameworks are expected: it doesn’t provide more features than JSON-RPC in a 100% Python environment.

Like the JSON-RPC transport, it depends on an external library, jsonrpclib-pelix which has no transient dependency.

Like most of the transport providers, this one is split in two components: the exporter and the importer. Both must be instantiated manually.

The exporter instance can be configured with the following property:

Property Default value Description
pelix.http.path /JABSORB-RPC The path to the JABSORB-RPC exporter servlet

To use this transport provider, you’ll need to install the following bundles and instantiate the associated components:

# Start the HTTP service with default parameters
install pelix.http.basic
start $?
instantiate pelix.http.service.basic.factory httpd

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Install and start the JABSORB-RPC importer and exporter with the default
# parameters
install pelix.remote.transport.jabsorb_rpc
start $?
instantiate pelix-jabsorbrpc-exporter-factory jabsorbrpc-exporter
instantiate pelix-jabsorbrpc-importer-factory jabsorbrpc-importer

MQTT discovery and MQTT-RPC Transport

Bundle:pelix.remote.discovery.mqtt, pelix.remote.transport.mqtt_rpc
Factories:pelix-remote-discovery-mqtt-factory, pelix-mqttrpc-exporter-factory, pelix-mqttrpc-importer-factory
Requires:nothing (everything goes through MQTT messages)
Libraries:paho

Finally, the MQTT discovery and transport protocols have been developped as a proof of concept with the fabMSTIC fablab of the Grenoble Alps University.

The idea was to rely on the lightweight MQTT messages to provide both discovery and transport mechanisms, and to let them be handled by low-power devices like small Arduino boards. Mixed results were obtained: it worked but the performances were not those intended, mainly in terms of latencies.

Those providers are kept in Pelix/iPOPO as they work and provide a non-HTTP way to communicate, but they won’t be updated without new contributions (pull requests, ...).

They rely on the Eclipse Paho library, previously known as the Mosquitto library.

The discovery instance can be configured with the following properties:

Property Default value Description  
mqtt.host localhost Host of the MQTT server
mqtt.port 1883 Port of the MQTT server
topic.prefix pelix/{appid}/remote-services Prefix of all MQTT messages (format string accepting the appid entry)
application.id None Application ID, to allow multiple applications on the same server

The transport exporter and importer instances should be configured with the same mqtt.host and mqtt.port properties as the discovery service.

To use the MQTT providers, you’ll need to install the following bundles and instantiate the associated components:

# Install Remote Services Core
install pelix.remote.registry
start $?
install pelix.remote.dispatcher
start $?

# Install and start the MQTT discovery and the MQTT-RPC importer and exporter
# with the default parameters
install pelix.remote.discovery.mqtt
start $?
instantiate pelix-remote-discovery-mqtt-factory mqttrpc-discovery

install pelix.remote.transport.mqtt_rpc
start $?
instantiate pelix-mqttrpc-exporter-factory mqttrpc-exporter
instantiate pelix-mqttrpc-importer-factory mqttrpc-importer

API

Endpoints

ExportEndpoint objects are created by transport providers and stored in the registry of the exports dispatcher. It is used by discovery providers to create a description of the endpoint to send over the network and suitable for the import-side.

class pelix.remote.beans.ExportEndpoint(uid, fw_uid, configurations, name, svc_ref, service, properties)

Represents an export end point (one per group of configuration types)

Parameters:
  • uid – Unique identified of the end point
  • fw_uid – The framework UID
  • configurations – Kinds of end point (xmlrpc, ...)
  • name – Name of the end point
  • svc_ref – ServiceReference of the exported service
  • service – Instance of the exported service
  • properties – Extra properties
Raises:

ValueError – Invalid UID or the end point exports nothing (all specifications have been filtered)

get_properties()

Returns merged properties

Returns:Endpoint merged properties
make_import_properties()

Returns the properties of this endpoint where export properties have been replaced by import ones

Returns:A dictionary with import properties
rename(new_name)

Updates the endpoint name

Parameters:new_name – The new name of the endpoint
configurations

Configurations of this end point

framework

Framework UID

instance

Service instance

name

Name of the end point

reference

Service reference

specifications

Returns the exported specifications

uid

End point unique identifier

ImportEndpoint objects are the description of an endpoint on the consumer side. They are given by the imports registry to the transport providers on the import side.

class pelix.remote.beans.ImportEndpoint(uid, framework, configurations, name, specifications, properties)

Represents an end point to access an imported service

Parameters:
  • uid – Unique identified of the end point
  • framework – UID of the framework exporting the end point (can be None)
  • configurations – Kinds of end point (xmlrpc, ...)
  • name – Name of the end point
  • specifications – Specifications of the exported service
  • properties – Properties of the service
configurations

Kind of end point

framework

UID of the framework exporting this end point

name

Name of the end point

properties

Properties of the imported service

specifications

Specifications of the service

uid

End point unique identifier

Core Services

The exports dispatcher service provides the pelix.remote.dispatcher (constant string stored in pelix.remote.SERVICE_DISPATCHER) service, with the following API:

class pelix.remote.dispatcher.Dispatcher

Common dispatcher for all exporters

get_endpoint(uid)

Retrieves an end point description, selected by its UID. Returns None if the UID is unknown.

Parameters:uid – UID of an end point
Returns:An ExportEndpoint or None.
get_endpoints(kind=None, name=None)

Retrieves all end points matching the given kind and/or name

Parameters:
  • kind – A kind of end point
  • name – The name of the end point
Returns:

A list of ExportEndpoint matching the parameters

The import registry service provides the pelix.remote.registry (constant string stored in pelix.remote.SERVICE_REGISTRY) service, with the following API:

class pelix.remote.registry.ImportsRegistry

Registry of discovered end points. End points are identified by their UID

add(endpoint)

Registers an end point and notifies listeners. Does nothing if the endpoint UID was already known.

Parameters:endpoint – An ImportEndpoint object
Returns:True if the end point has been added
contains(endpoint)

Checks if an endpoint is in the registry

Parameters:endpoint – An endpoint UID or an ImportEndpoint object
Returns:True if the endpoint is known, else False
lost_framework(uid)

Unregisters all the end points associated to the given framework UID

Parameters:uid – The UID of a framework
remove(uid)

Unregisters an end point and notifies listeners

Parameters:uid – The UID of the end point to unregister
Returns:True if the endpoint was known
update(uid, new_properties)

Updates an end point and notifies listeners

Parameters:
  • uid – The UID of the end point
  • new_properties – The new properties of the end point
Returns:

True if the endpoint is known, else False