Service development and testing

Our development teams build applications for artists creating visual effects through to management teams coordinating productions. A service-based architecture was chosen to provide consistent interfaces across the many different environments where this is required. We provide an ecosystem where developers of any level can safely write a service (a set of instructions regarding a specific topic) that are presented to developers and technical artists globally. To write a service the developer doesn't need any knowledge in regards to building large concurrent systems. The service is implemented through a simple Python API and the provided ecosystem allows services to exist in a standalone manner. The service concept was separated from the platform hosting it. This allows hosting in any application that provides a standard container (a service provider). Extracting this allowed for more rigorous and simple testing of services; it also allows developers to provide fake versions of their services publicly against which client code can be tested. The adage ʻeverything as a serviceʼ was applied to the development of internal facilities. This includes our management tools and the developer console, which presents the documentation of services and methods available to developers through a web interface. Infrastructure services were introduced to present an interface to facilities provided to a regular service, for example databases, configuration and centralized logging. Services can call other services and, similarly to infrastructures, services can be replaced with different services depending on the configuration. Services are exposed to a service (or client as we will see later) via a service provider just like in applications. Setting services up with the above patterns allows developers to iterate quickly and to include services within testing frameworks. It has also provided a standardized form across projects allowing developers to support and add to unfamiliar code easily. And last but not least it has given us full abstractions at all levels, users of services do not need to know the code underneath the hood be it at a service level or at an infrastructure level.

Building the cluster

Rather than building a single system, the new architecture defines a set of building blocks for constructing a distributed service platform. These can provide adapters for best of breed third party tools or, where necessary, custom implementations of functionality. Configuration is used to determine the number and types of modules to use and the parameters with which to initialize them. This allows the same platform to be used for small instances at a developerʼs desk up to a production environment of many nodes. The design enables improved components to be swapped into the existing system whilst forming the basis for an entirely new design.

Most practical applications require the service provider to handle multiple requests at the same time. Amanda provides a set of interchangeable concurrency modules. This allows the most appropriate Python model for parallel processing to be chosen. For work involving heavy I/O work we choose approaches that avoid waiting for the GIL, for example multiple processes and greenlets/coroutines, whilst for CPU bound work we can use threads which may prove more performant. Having the option to choose between mechanisms is important since there is not a solution that neatly fits all use cases. A pluggable concurrency abstraction also allows integration of new libraries as they become available. In future this might include the new asyncio (formerly Tulip) core library for Python 3.3+.

To benefit from concurrency, resource pooling, caching etc. we don't always want to execute the service locally to the service provider. Service proxies implement this behavior; they take the service, method and arguments of a request as their input and return the result. The proxy should be transparent to the service and service provider components. By chaining proxies, complex routing schemes can be built or analysis performed on the results returned. Some similarity can be drawn with middle-ware in the Web Services Gateway Interface (WSGI) specification. Communication between proxy and service provider is served by the transport. This abstraction provides an asynchronous interface to underlying technologies – Current implementations include queue based AMQP, ontop of RabbitMQ, and ØMQ and more naïve communications with standard UDP and TCP sockets. Most transports define both client and server parts of the system – however some, particularly HTTP-based transports, are designed to accept requests directly from external clients. Requests from external applications commonly use XMLRPC, JSONRPC or straight JSON. Transport implementations can be interchanged without impacting other components of Amanda or service developers.

In production, a request gateway implemented as a WSGI application fronts the HTTP protocols. Using the standard web components NGINX and μWSGI we can build a very scalable front end which internally uses the service provider, proxy, transport pattern to offload the requests to a backend. The gateway can also provide standard web facilities such as template rendering (through the Jinja2 library1) for general web clients. The gateway was a requirement as requests originate from applications written in many languages including C++, Python, JavaScript and domain specific languages such as mel. For us it was important that the client used across all those languages was a proven standard and lightweight. Most requests are served in near realtime (6ms round trip times) and are presented to the client in a synchronous way so using a frontend that supports a large number of HTTP like protocols allowed us to keep the clients simple and present the platform to an extremely wide variety of languages. Additionally, through the frontend, we can render a web page and present that directly if the requests was made from a browser.

The final behavior of the platform is defined in configuration. This allows the platform to be tuned to suit the work that a particular service is performing (I/O vs CPU bound). It is important to remember that every single component mentioned above be it the concurrency, transport, proxies or frontend can be changed, removed, updated without it impacting the service, the developer or any of the other components that make up the platform.

Also important to mention that internally and externally everything is a queue and presented as a queue. Going from the client to the frontend there is a queue, from the frontend onto the backend there is a queue etc. all the way down to a request being read of the transport and stored inside a queue until a concurrency object is ready to handle the request with the service provider.

This is where we think our platform might take a different approach. Rather than building the platform on top of a single great technology we didn't want to limit ourselves and be able to use all the other great technologies out there. There is no perfect solution for all problems but allowing to fine tune the platform according to different problems. The setup can now evolve in line with technological advancements and changes to the industry.

Maintenance and Monitoring (5 mins)

We will walk through how we are using the same setup with services, service providers, proxies and transports to manage clusters around the globe. Once again for our maintenance and monitoring we made sure everything is done as a service so that if there is a better tool in the future we could adopt it.

Through leveraging the configuration management and remote execute platform Salt, a new cluster can now be provisioned quickly. Management is itself provided as a service. Through this system, the current state is available and configuration changed across all servers globally. This has reduced routine maintenance tasks from a half day to a five-minute task, with less chance of human error. Monitoring and introspection are provided, as a service, to aid in day-to-day support, tuning and to help support analysis for future development.

Developers of services can trace requests from when they enter the system, producing a report of the sequence of methods being called, with the supplied arguments. For each call the time spent to fulfill each request is presented. Care was taken to minimize the impact of this on return result of the request. Due to everything being a queue we can collect the metrics after the result has been put back onto the transport and send to the user and thus minimize the impact of this collection on returning the result of the request This means that there is no requirement to put the system into a debug mode in order to obtain execution metrics.

With logging being a service we can dynamically change the logging configuration on a per service basis by making a request to the logging service taking away the need of changing configuration and restarting the service which often means a problem might have disappeared due to the reset.

Future/Conclusion (1 min)

Whilst developing the new generation of the platform there have been a number of possible applications that have emerged. The way in which we are able to scale the system would be suitable to run in a cloud environment – especially with the improvements to management allowing new nodes to be provisioned quickly. The ease of writing and integrating new components would allow integration with infrastructure provided by third-party cloud vendors. Other areas of interest include a smaller version of the platform running locally on a userʼs workstation and services for management of generic processes.

Main technologies and libraries currently used:

• Threading
• Gevent
• Eventlet
• Multiprocessing
• ZeroMQ
• RabbitMQ
• uwsgi
• Flask
• Salt
• nginx