Discussions

External Integration Needs

Hey, I wanted to post about this as I think it's an important point to bring up.

My firm mostly deals with integrations between software applications. Specifically, we've decided to focus on how to integrate Ubercart with SAP Business One and a few other selected back-end systems. This brings up another philosophical issue that should be addressed.

System Integration - Back end Rules

I know I'm going to catch a bit of a debate from this, but Ubercart is not the primary focus of most businesses - especially in the SME space. Here is the deal:


  • Few businesses are pure play ecommerce - those that are typically have separate accounting systems / SCM systems / CRM systems that are used in parallel

  • The vast majority of business rules are determined by back end software that is not customer facing.


    • Most SKU / Product ID systems come from some back-end software.

    • Pricing is almost always determined by the capabilities of an accounting system

    • Shipping and packaging capabilities are typically determined by warehouse management and logistics centers

    • Invoicing and tax requirements are driven by accounting software

    • You get the idea...


  • Integration is supported by most other major ecommerce packages - especially Magento and Miva. There are a ton of proprietary market solutions too for Quickbooks, SAP, Sage, Netsuite, etc.

  • There is a market demand for this - in just casual research Orchestra has found that there are many companies willing to put both time and money into projects that integrate their back end systems to Ubercart.

IMO - this is an essential step for Ubercart to reach the top of the ecommerce pile. Fundamentally, it comes down to the business cases that support Ubercart. Right now, Ubercart is a good ecommerce system, but, speaking from some experience, it's a pain to use the API to integrate with other systems.

So how do we solve this?

My first thought is - a more encompassing API would rock. I have some ideas of what this would entail, and here is why. Most of the ways of connecting with Ubercart fall along these lines:

* Direct database connections
* Web services

Direct DB connections suck because you always stand a chance of really screwing something up. Plus, it requires an underlying knowledge of how the DB operates in order to be effective at doing this. Personally, it's a last resort for us when talking integration.

Web services rock, but just doing request/response stuff is cumbersome. Often an integration needs to do more than just play with data - we actually need access to the underlying objects and functions so we can play around with them. Hence an API. Ubercart has this.

As far as I can tell in my examination of the Ubercart API, it's good, but it's still difficult to work with when we're talking about integration. There are a few reasons for this:

* Lack of true objects for products, carts, orders, customers, etc. in a lot of cases.
* Confusion about the use / application of hooks in many modules (i.e. shipping, packaging, pricing, donations, non-physical goods, etc).

Creation of a robust external API?

Ideally, it would be awesome to have a robust API that could take the place of something like services, features, services, feeds and node import for integration. This is a holy grail, and isn't that well formed in my mind as to what the end product would actually look like.

* So, how do we do this?
* What specifically do other people need? (I've got a lot of info from our devs regarding integration needs, but nothing else really)
* How does this get built / what does it look like?

Some Ideas (Brainstorming)


  • Documentation improvements - code examples that are used for illustration and show how to really use the API for good (as opposed to evil). This would have some step-by-step examples that show good coding standards.

  • It would rock to have UC services - web services specifically designed for Ubercart

  • More accessible functions for the areas where Ubercart is lacking some core functionality (tax, shipping, etc). I don't want to end up replacing modules for UC, but it needs some improvement.

I'm really curious to hear what you guys think.

chrisstrahl
Posted: Nov 25, 2009

Comments

Ryan Ryan Szrama on November 25, 2009

Hey Chris, thanks for starting this discussion off so well! I think you've perfectly summarized a lot of my thoughts on the topic, and I'll add a small touch of my vision.

Basically, I think you're right on. The major e-commerce sites out there don't need an all encompassing system. Their websites are their initial point of contact with the customer, but the data is never meant to remain and be administered solely through the website. It's basically a data collection point with payment, accounting, fulfillment, and analytics all happening off-site.

The way Ubercart was developed caters to what has been the principle base of contributors and users thus far. So, the fact that we have a usable e-commerce system is a good starting point. However, as Chris has pointed out (somewhat more politely than I probably feel at liberty to do), the APIs in Ubercart just aren't really up to snuff for painless integration with third party services. I spend a significant amount of time for my clients wrestling with it to do third party services integration as well and find myself consistently having to re-implement my design patterns, objects, queues and such... although I did finally spin off my export queues into http://drupal.org/project/export_queue.

My vision for Ubercore is for it to operate as a conduit. It's the principle data gathering point, and it's designed with easy transmission of that data in mind. The data should still be available and usable on the site itself, but there needs to be more attention paid in the future to getting that data out. Chris has balanced out my thinking by bringing up the point that we need a good way to synchronize data in as well. Importing and updating products from external sources has been a consistent pain point, but I'm still really hoping that we can outsource that to other Drupal modules designed with importing entity data as well. Perhaps the same will be possible for exporting?

Remember, products and orders will be entities moving forward, so I'm hoping that with the consistent underlying structure, modules will pop up that will import / export any entity type... so the module could handle nodes and users as easily as products and orders.

redben on November 26, 2009

I was just about to post something similar, but you exposed it so well !
One more feature that emphasizes on the need for a rock solid api, totally independent from drupal ui !

kulvik on January 25, 2010

It's great to see this topic discussed. My company make a living from delivering custom UC stores integrated with third-party systems. We deliver tailored solutions for some of the biggest online stores in Norway and they all have one thing in common: They are, in some way or another, integrated with one or more underlying systems. However, the role of UC varies greatly depending on the capabilities (and cost) og the underlying systems. Let me take a couple of examples:

www.akademika.no
1,4+ million products, 5+ million nodes. 30.000 active customers.
Integrated with M3 from Lawson (http://www.lawson.com/wcw.nsf/pub/bpr).
Currently we have a file-based (XML) integration that integrates customers, orders (and changes on orders on item-level) and tracking information for packages. We also have real-time stock values via web services.
This is a typical case. M3 is the big and expensive system the organisation dependes on, but it's virtually impossible to make changes in and the costs related to changes in this system is just not right :P The result is that we are building more and more functionality in UC and moving further away from M3 and M3s core systems all the time.

www.nrkbutikken.no
This is the main television broadcasting company in norway (like BBC in UK). In this store we have the following integrations:
Stock: Visma Global. File integration (XML).
Accounting: Formula. File integration (XML).
Digital products: Paragallo (paragallo.com). Webservice integration.
Here UC is in the middle of things, just making sure the others systems are getting what they need.

I could list a lot more here, but the conclusion would be the same for all of them. The most important thing (at least for us) is making sure UC has a good internal API (not external) for getting customers, orders, changes on orders, payment/shipping-methods/data, tracking numbers etc. in and out of UC.

We have been doing this for some years now and we've never gotten to a situation where the third-party applications need to integrate themselves with UC (throught webservices or anything else). It's always the other way around. The reason is of course that the underlying ERP/CRM systems are both difficult and expensive to tailor compared to Drupal/UC. It's almost always easier and cheaper for the customer to do the "dirty" work in UC. Therefore I don't see the big need for an external API with webservices (at least not right now). I'd much rather see a good internal API that makes it easy to import/export as many different entities of übercart as possible.

This is of course just our experience and needs i'm writing about here. Others may not agree based on other experiences.

If this ends up in a module or more solid discussions we'd like to contribute to it in any way we can.

Best regards,
Thomas Kulvik

Ny Media AS
www.nymedia.no
+47 4000 7955

markalosey on April 9, 2010

My team is willing to participate in any way we can. We are running SAP ECC. We have used a variety of approaches to integration that are all really hacked together solutions at this point.

Stock: File Integration, imported directly to db via drush cron script
Pricing: Same story
Orders: In process....

Currently, in our development environment we have rolled up a more flexible solution to integration with SAP. It's not particularly specific to Ubercart. (on purpose) But it's not particularly specific to SAP either. It's a middleware concept with a abstracted order source and an abstracted order destination (with portal for manual management).

I could see this concept expanding to include other backend systems and also handling other order sources. (3rd party shipping etc..)

I know that project is a little out of scope for this discussion, but I bring it up as a demonstration of the real need for a more robust 'connection' to ubercart/drupal commerce.

I am on board and can bring myself and my team into the effort.

Other necessary features our business needs to address still as a multi-channel merchant are:

  • Provide order history regardless of sales channel
  • Provide shipment tracking regardless of sales channel

FX Trade Online... (not verified) on April 4, 2011

There are many examples of enterprises that start their integration journey with an ESB and then move up to a more sophisticated integration suite as their needs grow. This is not a radical approach. The actual migration is often minimized as the providers of ESB and CIS products are often the same vendors (who have pre-integrated the related tools).