DISTRIB HAS MOVED!

Distrib has moved over to GitHub, this is mainly because of superior wiki functionality, Git not throwing an utter fit if I don't have internet access as well as a host of other factors.

You can find Distrib on github over at: https://github.com/clintkpearson/Distrib

Project Description

Distributed Processing Grid (Distrib) aims to provide developers with a simple, easy-to-use framework for enabling distributed processing within their applications; from the smallest of requirements to large processing clusters, Distrib aims to be of use.

What makes up Distrib?

While Distrib is a broad framework for enabling developers to create distributed processing clusters, it primarily consists of:

  • A .NET library
    • This is the core Distrib library and contains everything you need to utilise Distrib
  • Accompanying tools
    • These range from simple applications for quick and dirty help with testing and development, through to applications you can ship with your own to support your users in working with the cluster
      • Many of the applications for interacting with the Distrib cluster are default implementations and may not be suitable for your particular shipping needs, you can always implement your own to any exacting specifications, all the code and help is there.

What are the core features of Distrib?

While Distrib is still in development many features may not be currently implemented or are incomplete, nevertheless, the following (by no means exhaustive) list aims to provide a glimpse into the planned capability:

  • Fault Resistant
    • At its core most of Distrib is designed to be separated by AppDomain boundaries, helping to minimise the risk of a rogue process / exception bringing down a node in a cluster
  • Simple & Easy to use
    • Core Distrib components are designed to follow common patterns and expose intuitive interfaces, allowing you to express what you want to achieve instead of having to hunt for a particular class.
  • Controllable & Extensible
    • Distrib contains a lot of default implementation and behaviour to enable as much “out-of-the-box” functionality as possible, most of this can be hooked into or passed custom implementation allowing you to have a say over the system.
  • Intelligent
    • What can intelligence mean for a distributed processing cluster? How about:
      • Load balancing
        • Jobs shuttled around the cluster based on current load and resources
      • Instance management
        • By monitoring the status of the cluster, Distrib can spin up or spin down process instances on nodes as needed; this is great for expanding capability during periods of high load and reducing resource consumption
      • Fit-for-purpose analysis
        • Some processes are going to be more resource intensive than others and this may fluctuate with varying input data; by monitoring how resources are consumed and weighed against node capabilities (e,g. hardware), Distrib can identify likely candidate nodes for certain jobs, thus enabling the right hardware to be selected as needed.
  • Process versioning
    • Processes are likely to change over time, pulling down a whole node or cluster for the sake of a bug-fix is maddening in a production environment; Distrib processes are versioned and job execution is closely tracked to process identity, this enables updated processes to be pushed into the cluster and for different upgrade strategies to be employed.
  • Delayed data access
    • When executing a job, processes can be passed input values that have already been set, or can request the data through value providers; value providers enable the core input system to still be used (along with benefits such as caching etc), but enables data access from databases and web-services  (for example)
  • Pluggable communication
    • Distrib utilises its own communication methods for network calls, but the communications method can be swapped out if required.

Last edited Apr 29, 2013 at 2:31 AM by cpearson1990, version 19